[
  {
    "path": ".cargo/config",
    "content": "[build]\n# Postgres symbols won't ve available until runtime\nrustflags = [\"-C\", \"link-args=-Wl,-undefined,dynamic_lookup\"]\n"
  },
  {
    "path": ".dockerignore",
    "content": "**/*.iml\n**/*.o\n**/.DS_Store\n.editorconfig\n.idea\n.vscode\n.vsls.json\n.git\nold-versions\ntarget\ntarget-analyzer\n"
  },
  {
    "path": ".git-blame-ignore-revs",
    "content": "# Merge and parent commit for cargo fmt changes\nb7433344f90b142094e73e84c332385498db9335\n8b50127c9e4bad1696a68a800ce1ef019cf6fc3c\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/bug-report.md",
    "content": "---\nname: Bug Report\nabout: Something is not working as expected\ntitle: ''\nlabels: bug\nassignees: ''\n\n---\n\n**Relevant system information:**\n - OS: [e.g. Ubuntu 16.04, Windows 10 x64, etc]\n - PostgreSQL version (output of `SELECT version();`): [e.g. 12.0, 13.2, etc]\n - TimescaleDB Toolkit version (output of `\\dx timescaledb_toolkit` in `psql`): [e.g. 1.0.0]\n - Installation method: [e.g., \"Timescale Cloud\", \"docker\", \"source\"]\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**To Reproduce**\nSteps to reproduce the behavior:\n1. Go to '...'\n2. Click on '....'\n3. Scroll down to '....'\n4. See error\n\n**Expected behavior**\nA clear and concise description of what you _expected_ to happen.\n\n**Actual behavior**\nA clear and concise description of what _actually_ happened.\n\n**Screenshots**\nIf applicable, add screenshots to help explain your problem.\n\n**Additional context**\nAdd any other context about the problem here.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature-request.md",
    "content": "---\nname: Feature Request\nabout: Suggest an idea for this project\ntitle: ''\nlabels: feature-request\nassignees: ''\n\n---\n\n**Is your feature request related to a problem? Please describe.**\nA clear and concise description of what the problem is. Ex. I'm always frustrated when [...]\n\n**Describe the solution you'd like**\nA clear and concise description of what you would like to happen.\n\n**Describe alternatives you've considered**\nA clear and concise description of any alternative solutions or features you've considered.\n\n**Additional context**\nAdd any other context or screenshots about the feature request here.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature-stabilization.md",
    "content": "---\nname: Feature Stabilization\nabout: Checklist of tasks to move a feature out of experimental\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\n## [\\<Feature Name>](<link to root issue for feature>)\n\n**What evidence do we have the feature is being used**\n\n**Why do we feel this feature is ready to be stable**\n\n**Is there any known further work needed on this feature after stabilization**\n\n**Are there any compatibility concerns that may arise during future work on this feature**\n\n### Feature History\n- Experimental release version:\n- Last version modifying on-disk format:\n- Target stabilization version:\n\n\n### Stabilization checklist:\n- [ ] Ensure tests exist for all public API\n- [ ] Ensure API documentation exists and is accurate\n- [ ] Remove `toolkit_experimental` tags and update test usages\n- [ ] Add arrow operators for accessors if applicable\n- [ ] Ensure arrow operators have test coverage\n- [ ] If present, ensure `combine` and `rollup` are tested\n- [ ] Add serialization tests for on disk format\n- [ ] Add upgrade tests\n- [ ] Add continuous aggregate test\n- [ ] Add feature level documentation\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/proposed-feature.md",
    "content": "---\nname: Proposed Feature\nabout: Propose a solution to a problem or wishlist item\ntitle: ''\nlabels: proposed-feature\nassignees: ''\n\n---\n\n## What's the functionality you would like to add ##\nA clear and concise description of what you want to happen.\n\n## How would the function be used ##\nGive an example of what a workflow using the function would look like\n\n## Why should this feature be added?  ##\nA clear and concise description of what the problem is. Ex. I'm always frustrated when [...]\nIs your feature request related to a problem? A wishlist item?\n\n### What scale is this useful at? ###\nIs this useful for large data sets? Small ones? Medium sized?\n\n## Drawbacks ##\nAre there any issues with this particular solution to the problem?\n\n## Open Questions ##\nAre any questions we'd need to address before releasing this feature?\n\n## Alternatives ##\nAre there any alternative to the solutions chosen in the above text? Are there any other issues competing with this one?\n"
  },
  {
    "path": ".github/workflows/add-to-bugs-board.yml",
    "content": "name: Add bugs to bugs project\n\n\"on\":\n  issues:\n    types: [opened, labeled]\n  issue_comment:\n    types: [created, edited]\n\njobs:\n  add-to-project:\n    name: Add issue to project\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/add-to-project@v1.0.2\n        with:\n          project-url: https://github.com/orgs/timescale/projects/55\n          github-token: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n\n\n  waiting-for-author:\n    name: Waiting for Author\n    runs-on: ubuntu-latest\n    if: github.event_name == 'issues' && github.event.action == 'labeled'\n      && github.event.label.name == 'waiting-for-author'\n    steps:\n      - uses: leonsteinhaeuser/project-beta-automations@v2.2.1\n        with:\n          gh_token: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n          organization: timescale\n          project_id: 55\n          resource_node_id: ${{ github.event.issue.node_id }}\n          status_value: 'Waiting for Author'\n\n  waiting-for-engineering:\n    name: Waiting for Engineering\n    runs-on: ubuntu-latest\n    if: github.event_name == 'issue_comment' && !github.event.issue.pull_request\n      && contains(github.event.issue.labels.*.name, 'waiting-for-author')\n    steps:\n      - name: Check if organization member\n        uses: tspascoal/get-user-teams-membership@v3\n        id: checkUserMember\n        with:\n         username: ${{ github.actor }}\n         organization: timescale\n         team: 'database-eng'\n         GITHUB_TOKEN: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n      - name: Remove waiting-for-author label\n        if: ${{ steps.checkUserMember.outputs.isTeamMember == 'false' }}\n        uses: andymckay/labeler@3a4296e9dcdf9576b0456050db78cfd34853f260\n        with:\n          remove-labels: 'waiting-for-author, no-activity'\n          repo-token: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n      - name: Move to waiting for engineering column\n        if: ${{ steps.checkUserMember.outputs.isTeamMember == 'false' }}\n        uses: leonsteinhaeuser/project-beta-automations@v2.2.1\n        with:\n          gh_token: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n          organization: timescale\n          project_id: 55\n          resource_node_id: ${{ github.event.issue.node_id }}\n          status_value: 'Waiting for Engineering'\n"
  },
  {
    "path": ".github/workflows/ci.yml",
    "content": "name: CI\non:\n  pull_request:\n  push:\n    branches:\n    - main\n    - staging\n    - trying\n  schedule:\n    # TimescaleDB integration: 8am UTC, 3am Eastern, midnight Pacific\n    - cron: '0 8 * * 1-4'\n    # Testing on every platform: 6am UTC, 1am Eastern, 10pm Pacific\n    - cron: '0 6 * * 1-4'\n  workflow_dispatch:\n    inputs:\n      container-image:\n        description: 'Container image to pull from DockerHub'\n        required: false\n      tsdb-commit:\n        description: 'TimescaleDB commit to use'\n        default: ''\n        required: false\n      tsdb-repo:\n        description: 'TimescaleDB repo to use'\n        default: 'https://github.com/timescale/timescaledb.git'\n        required: false\n      all-platforms:\n        description: 'Test all platforms'\n        type: boolean\n        default: false\n\njobs:\n  testpostgres:\n    name: Test Postgres\n    runs-on: ${{ contains(matrix.container.image, 'arm64') && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}\n    container:\n      image: ${{ inputs.container-image || 'timescaledev/toolkit-builder-test' }}:${{ matrix.container.image }}\n    strategy:\n      fail-fast: false\n      max-parallel: 12\n      matrix:\n        pgversion: [15, 16, 17, 18]\n        container:\n        - os: rockylinux\n          version: \"9\"\n          image: rockylinux-9-x86_64\n          schedule: true\n        - os: debian\n          version: \"13\"\n          image: debian-13-arm64\n          schedule: true\n        - os: debian\n          version: \"13\"\n          image: debian-13-amd64\n          schedule: true\n        - os: debian\n          version: \"12\"\n          image: debian-12-arm64\n          schedule: ${{ inputs.all-platforms || ( github.event_name == 'schedule' && github.event.schedule == '0 6 * * 1-4' ) }}\n        - os: debian\n          version: \"12\"\n          image: debian-12-amd64\n          schedule: ${{ inputs.all-platforms || ( github.event_name == 'schedule' && github.event.schedule == '0 6 * * 1-4' ) }}\n        - os: debian\n          version: \"11\"\n          image: debian-11-amd64\n          schedule: ${{ inputs.all-platforms || ( github.event_name == 'schedule' && github.event.schedule == '0 6 * * 1-4' ) }}\n        - os: ubuntu\n          version: \"24.04\"\n          image: ubuntu-24.04-amd64\n          schedule: true\n        - os: ubuntu\n          version: \"22.04\"\n          image: ubuntu-22.04-amd64\n          schedule: ${{ inputs.all-platforms || ( github.event_name == 'schedule' && github.event.schedule == '0 6 * * 1-4' ) }}\n        exclude:\n        - container:\n            skip: true\n        - container:\n            schedule: false\n    env:\n      # TODO Why?  Cargo default is to pass `-C incremental` to rustc; why don't we want that?\n      #   https://doc.rust-lang.org/rustc/codegen-options/index.html#incremental\n      #   Well turning it off takes the extension target size down from 3G to 2G...\n      CARGO_INCREMENTAL: 0\n      # TODO Why?  If we're concerned about trouble fetching crates, why not\n      #  just fetch them once at the time we select a dependency?\n      #  Errors fetching crates are probably rare enough that we don't see the\n      #  need to bother, but then why not just let the build fail?\n      CARGO_NET_RETRY: 10\n      # TODO What reads this?  It's not listed on\n      #  https://doc.rust-lang.org/cargo/reference/environment-variables.html\n      CI: 1\n      RUST_BACKTRACE: short\n\n    steps:\n    - name: Checkout Repository\n      uses: actions/checkout@v5\n      with:\n        ref: ${{ github.event.pull_request.head.sha }}\n        token: ${{ secrets.GITHUB_TOKEN }}\n\n    # Github Actions provides a bind mounted working directory for us, where\n    # the above checkout happens, and where caches are read from and restored\n    # to, and it's all owned by 1001.  Our container image is `USER root` so\n    # we have no problem writing anywhere, but we run some things as user\n    # 'postgres', which used to be user 1000 but is now 1001.  Hoping in the\n    # future to make our container image `USER postgres` and further simplify\n    # this file and the packaging Actions file, but it's non-trivial.\n    - name: chown Repository\n      run: |\n        chown -R postgres .\n\n    - name: Build and install TimescaleDB\n      if: ${{ (github.event_name == 'schedule' && github.event.schedule == '0 8 * * 1-4') || inputs.tsdb-commit != '' }}\n      run: ./tools/install-timescaledb '${{ matrix.pgversion }}' '${{ inputs.tsdb-repo || 'https://github.com/timescale/timescaledb.git' }}' '${{ inputs.tsdb-commit == '' && 'main' || matrix.tsdb_commit || inputs.tsdb-commit }}'\n\n    # TODO After the container image contains a primed target dir, is this still worth it?\n    #   Only possible advantage is this one is per-pg-version but what's the impact?\n    - name: Cache cargo target dir\n      uses: actions/cache@v4\n      if: ${{ matrix.container.image == 'debian-11-amd64' }}\n      with:\n        path: target\n        key: ${{ runner.os }}-test-pg${{ matrix.pgversion }}-target-${{ hashFiles('Cargo.lock', '.github/workflows/ci.yml') }}\n        restore-keys: ${{ runner.os }}-test-pg${{ matrix.pgversion }}-target-  \n\n    # Packages not \n    #\n\n    - name: Run pgrx tests\n      run: |\n        if [ \"${{ matrix.container.version }}\" = 7 ]; then\n          # needed for pgrx to find clang\n          set +e # will succeed but have non-zero exit code\n          . scl_source enable llvm-toolset-7\n          set -e\n        fi\n\n        su postgres -c 'sh tools/build -pg${{ matrix.pgversion }} test-extension 2>&1'\n\n    - name: Run doc tests\n      # depends on TSDB, which requires PG >=13\n      if: ${{ matrix.pgversion >= 13 && (matrix.pgversion >= 13 && (matrix.pgversion <= 15 || ((github.event_name == 'schedule' && github.event.schedule == '0 8 * * 1-4') || inputs.tsdb-commit != ''))) }}\n      run: su postgres -c 'sh tools/build -pg${{ matrix.pgversion }} test-doc 2>&1'\n\n    - name: Run binary update tests (deb)\n      # depends on TSDB, which requires PG >=13\n      if: ${{ (matrix.container.os == 'debian' || matrix.container.os == 'ubuntu') && (matrix.pgversion >= 13 && (matrix.pgversion <= 16 || ((github.event_name == 'schedule' && github.event.schedule == '0 8 * * 1-4') || inputs.tsdb-commit != ''))) }}\n      run: |\n        su postgres -c 'OS_NAME=${{ matrix.container.os }} OS_VERSION=${{ matrix.container.version }} tools/testbin -version no -bindir / -pgversions ${{ matrix.pgversion }} ci 2>&1'    \n    - name: Run binary update tests (EL)\n      if: ${{ (matrix.container.os == 'rockylinux') && matrix.container.version != '9' && (matrix.pgversion >= 13 && (matrix.pgversion <= 16 || ((github.event_name == 'schedule' && github.event.schedule == '0 8 * * 1-4') || inputs.tsdb-commit != ''))) }}\n      run: |\n        su postgres -c 'OS_NAME=${{ matrix.container.os }} OS_VERSION=${{ matrix.container.version }} tools/testbin -version no -bindir / -pgversions ${{ matrix.pgversion }} rpm_ci 2>&1'\n\n  testcrates:\n    name: Test Crates\n    runs-on: ubuntu-24.04\n    container:\n      image: ${{ inputs.container-image || 'timescaledev/toolkit-builder' }}:debian-11-amd64\n      env:\n        CARGO_INCREMENTAL: 0\n        CARGO_NET_RETRY: 10\n        CI: 1\n        RUST_BACKTRACE: short\n\n    steps:\n    - name: Checkout Repository\n      uses: actions/checkout@v5\n      with:\n        ref: ${{ github.event.pull_request.head.sha }}\n\n    - name: chown Repository\n      run: chown -R postgres .\n\n    - name: Cache cargo target dir\n      uses: actions/cache@v4\n      with:\n        path: target\n        key: ${{ runner.os }}-test-crates-target-${{ hashFiles('Cargo.lock', '.github/workflows/ci.yml') }}\n        restore-keys: ${{ runner.os }}-test-crates-target-\n\n    - name: Run Crates Tests\n      run: su postgres -c 'sh tools/build test-crates 2>&1'\n"
  },
  {
    "path": ".github/workflows/ci_image_build.yml",
    "content": "name: Build CI Image\n\non:\n  pull_request:\n    paths:\n      - 'docker/ci/**'\n      - '.github/workflows/ci_image_build.yml'\n      - 'tools/dependencies.sh'\n  workflow_dispatch:\n    inputs:\n      tag-base:\n        description: 'Push image to DockerHub with this base tag (remove \"-test\" enable)'\n        required: false\n        # Repeating the default here for ease of editing in the github actions form.  Keep in sync with below.\n        default: timescaledev/toolkit-builder-test\n      toolkit-commit:\n        description: 'Toolkit commit (branch, tag, etc.) to build image from'\n        required: false\n        default: main\n      builder-commit:\n        description: 'Commit (branch, tag, etc.) on release-build-scripts repository to use'\n        required: false\n\njobs:\n  build:\n    env:\n      GITHUB_TOKEN: ${{ secrets.ORG_AUTOMATION_TOKEN}}\n    runs-on: ubuntu-24.04\n    steps:\n      - name: Run release-build-scripts job\n        # Repeating the default here for 'pull_request'.  Keep in sync with above.\n        run: |\n          echo \"toolkit-commit: ${{ inputs.toolkit-commit || github.event.pull_request.head.sha }}\"\n          echo \"builder: ${{ inputs.builder-commit || 'main' }}\"\n          echo \"tag-base: ${{ inputs.tag-base || 'timescaledev/toolkit-builder-test' }}\"\n          gh workflow run toolkit-image.yml \\\n                -R timescale/release-build-scripts \\\n                -r ${{ inputs.builder-commit || 'main' }} \\\n                -f tag-base=${{ inputs.tag-base || 'timescaledev/toolkit-builder-test' }} \\\n                -f toolkit-commit=${{ inputs.toolkit-commit || github.event.pull_request.head.sha }}\n"
  },
  {
    "path": ".github/workflows/clippy_rustfmt.yml",
    "content": "name: Clippy and rustfmt\non:\n  pull_request:\n  push:\n    branches:\n    - main\n    - staging\n    - trying\n  workflow_dispatch:\n    inputs:\n      container-image:\n        description: 'Container image to pull from DockerHub'\n        required: false\n\njobs:\n  clippy:\n    name: Clippy/rustfmt Test\n    runs-on: ubuntu-24.04\n    container:\n      # Duplicated from ci.yml\n      image: ${{ inputs.container-image || 'timescaledev/toolkit-builder-test:debian-11-amd64' }}\n      env:\n        # TODO: See TODOs on duplicate block in ci.yml\n        CARGO_INCREMENTAL: 0\n        CARGO_NET_RETRY: 10\n        CI: 1\n        RUST_BACKTRACE: short\n\n    steps:\n    - name: Checkout Repository\n      uses: actions/checkout@v5\n      with:\n        ref: ${{ github.event.pull_request.head.sha }}\n\n    - name: chown Repository\n      run: chown -R postgres .\n\n    - name: Cache cargo target dir\n      uses: actions/cache@v4\n      with:\n        path: target\n        key: ${{ runner.os }}-clippy-target-${{ hashFiles('Cargo.lock', '.github/workflows/clippy_rustfmt.yml') }}\n        restore-keys: ${{ runner.os }}-clippy-target-\n\n    - name: Run Clippy\n      # Github captures stdout and stderr separately and then intermingles them\n      # in the wrong order.  We don't actually care to distinguish, so redirect\n      # stderr to stdout so we get the proper order.\n      run: su postgres -c 'sh tools/build clippy 2>&1'\n\n    - name: Verify formatting\n      run: su postgres -c 'cargo fmt --check 2>&1'\n"
  },
  {
    "path": ".github/workflows/dependency-updates.yml",
    "content": "name: Dependency Updates\non:\n  schedule:\n    # Run on the 1st of every month at 9:00 AM UTC\n    - cron: '0 9 1 * *'\n  workflow_dispatch:\n\nenv:\n  CARGO_TERM_COLOR: always\n\njobs:\n  update-dependencies:\n    runs-on: ubuntu-latest\n    steps:\n    - uses: actions/checkout@v5\n      with:\n        token: ${{ secrets.GITHUB_TOKEN }}\n\n    - uses: dtolnay/rust-toolchain@stable\n      with:\n        toolchain: stable\n\n    - name: Cache cargo registry and index\n      uses: actions/cache@v4\n      with:\n        path: |\n          ~/.cargo/registry/index/\n          ~/.cargo/registry/cache/\n          ~/.cargo/git/db/\n        key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}\n        restore-keys: |\n          ${{ runner.os }}-cargo-\n\n    - name: Update dependencies\n      run: |\n        # Update dependencies and capture the output\n        cargo update --verbose > update_output.txt 2>&1 || true\n        \n        # Check if Cargo.lock was modified\n        if git diff --quiet Cargo.lock; then\n          echo \"NO_UPDATES=true\" >> $GITHUB_ENV\n          echo \"No dependency updates available\"\n        else\n          echo \"NO_UPDATES=false\" >> $GITHUB_ENV\n          echo \"Dependencies updated, changes detected in Cargo.lock\"\n        fi\n\n    - name: Run cargo check\n      if: env.NO_UPDATES == 'false'\n      run: cargo check --all-targets --verbose\n\n    - name: Run tests\n      if: env.NO_UPDATES == 'false'\n      run: cargo test --verbose\n\n    - name: Generate update summary\n      if: env.NO_UPDATES == 'false'\n      run: |\n        echo \"## 📦 Monthly Dependency Updates\" > pr_body.md\n        echo \"\" >> pr_body.md\n        echo \"This PR contains automated dependency updates for $(date +'%B %Y').\" >> pr_body.md\n        echo \"\" >> pr_body.md\n        echo \"### Changes:\" >> pr_body.md\n        echo \"\\`\\`\\`\" >> pr_body.md\n        cat update_output.txt >> pr_body.md\n        echo \"\\`\\`\\`\" >> pr_body.md\n        echo \"\" >> pr_body.md\n        echo \"### Verification:\" >> pr_body.md\n        echo \"- ✅ \\`cargo check\\` passed\" >> pr_body.md\n        echo \"- ✅ \\`cargo test\\` passed\" >> pr_body.md\n        echo \"\" >> pr_body.md\n        echo \"---\" >> pr_body.md\n        echo \"*This PR was automatically created by the monthly dependency update workflow.*\" >> pr_body.md\n\n    - name: Create Pull Request\n      if: env.NO_UPDATES == 'false'\n      uses: peter-evans/create-pull-request@v6\n      with:\n        token: ${{ secrets.GITHUB_TOKEN }}\n        commit-message: \"chore: update dependencies for $(date +'%B %Y')\"\n        title: \"chore: Monthly dependency updates - $(date +'%B %Y')\"\n        body-path: pr_body.md\n        branch: dependency-updates/$(date +'%Y-%m')\n        delete-branch: true\n        labels: |\n          dependencies\n          automated\n        assignees: ${{ github.repository_owner }}\n        draft: false\n\n    - name: Summary\n      run: |\n        if [ \"$NO_UPDATES\" = \"true\" ]; then\n          echo \"✅ No dependency updates needed - all dependencies are up to date!\"\n        else\n          echo \"✅ Dependency update PR created successfully!\"\n        fi\n"
  },
  {
    "path": ".github/workflows/packaging.yml",
    "content": "# Trigger package workflows on release tagging\nname: Build packages\non:\n  push:\n    tags:\n    - \"[0-9]+.[0-9]+.[0-9]+\"\n  workflow_dispatch:\n\njobs:\n  package:\n    env:\n      GITHUB_TOKEN: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n    runs-on: ubuntu-24.04\n    steps:\n      - name: Set env\n        run: echo \"RELEASE_VERSION=${GITHUB_REF#refs/*/}\" >> $GITHUB_ENV\n\n      - name: Debian and Ubuntu packages\n        if: always()\n        run: |\n          gh workflow run toolkit-apt.yml -R timescale/release-build-scripts -r main -f version=${{ env.RELEASE_VERSION }} -f upload-artifacts=true\n\n      - name: RPM packages\n        if: always()\n        run: |\n          gh workflow run toolkit-rpm.yml -R timescale/release-build-scripts -r main -f version=${{ env.RELEASE_VERSION }} -f upload-artifacts=true\n\n"
  },
  {
    "path": ".github/workflows/release.yml",
    "content": "name: Release\non:\n  workflow_dispatch:\n    inputs:\n      version:\n        description: 'New version number for release'\n        required: true\n      commit:\n        description: 'Commit id to branch from (default is HEAD of main)'\n        type: string\n        required: false\n        default: main\n      # TODO Make this harder to screw up by making a checkbox.\n      dry-run:\n        description: '-n for dry-run, -push to really release'\n        type: string\n        required: false\n        default: -n\n\njobs:\n  release:\n    name: Release\n    runs-on: ubuntu-24.04\n    container:\n      image: timescaledev/toolkit-builder-test:debian-11-amd64\n\n    steps:\n    - name: Checkout Repository\n      uses: actions/checkout@v5\n      with:\n        ref: ${{ inputs.commit }}\n\n    - name: chown Repository\n      run: chown -R postgres .\n\n    - name: Install dependencies not yet in image\n      run: su postgres -c 'tools/release setup' 2>&1\n\n    - name: Run tools/release\n      env:\n        GITHUB_TOKEN: ${{ secrets.ORG_AUTOMATION_TOKEN }}\n        ACTIONS_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n      run: su postgres -c 'tools/release ${{ inputs.dry-run }} -version ${{ inputs.version }} ${{ inputs.commit }}' 2>&1\n"
  },
  {
    "path": ".github/workflows/report_packaging_failures.yml",
    "content": "name: Report Build Package Failures\non:\n  workflow_run:\n    workflows: [Build packages, Build CI Image, CI]\n    types: [completed]\n\njobs:\n  on-failure:\n    runs-on: ubuntu-24.04\n    if: ${{ github.event.workflow_run.conclusion != 'success' && github.event.workflow_run.event != 'pull_request' }}\n    steps:\n      - name: slack-send\n        uses: slackapi/slack-github-action@v1.19.0\n        with:\n          payload: |\n            {\n              \"blocks\": [\n                {\n                  \"type\": \"section\",\n                  \"text\": {\n                    \"type\": \"mrkdwn\",\n                    \"text\": \"Workflow run <${{ github.event.workflow_run.html_url }}|${{ github.event.workflow.name}}#${{ github.event.workflow_run.run_number }}>\"\n                  }\n                },\n                {\n                  \"type\": \"section\",\n                  \"fields\": [\n                    {\n                      \"type\": \"mrkdwn\",\n                      \"text\": \"*Status*\\n`${{ github.event.workflow_run.conclusion }}`\"\n                    },\n                    {\n                      \"type\": \"mrkdwn\",\n                      \"text\": \"*Triggered By*\\n<${{ github.event.sender.html_url }}|${{ github.event.sender.login }}>\"\n                    }\n                  ]\n                }\n              ]\n            }\n        env:\n          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_NOTIFY_WEBHOOK_URL }}\n          SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK\n"
  },
  {
    "path": ".gitignore",
    "content": ".DS_Store\n/.idea\n/.vscode\n/.vsls.json\n/old-versions\n/target\n*.iml\n/target-analyzer\n/.editorconfig\n"
  },
  {
    "path": "Cargo.toml",
    "content": "[workspace]\nresolver = \"2\"\n\nmembers = [\n    \"crates/t-digest-lib\",\n    \"extension\",\n    \"tools/post-install\",\n    \"tools/sql-doctester\",\n    \"tools/update-tester\",\n]\n\n[profile.release]\nlto = \"fat\"\ndebug = true\ncodegen-units = 1\n"
  },
  {
    "path": "Changelog.md",
    "content": "# Toolkit Changelog\n\n## Process for updating this changelog\n\nThis changelog should be updated as part of a PR if the work is worth noting (most of them should be). If unsure, always add an entry here for any PR targeted for the next release. It's easier to remove than add an entry at final review time for the next release.\n\n## Next Release (Date TBD)\n\n#### New experimental features\n\n#### Bug fixes\n\n#### Other notable changes\n\n#### Shout-outs\n\n**Full Changelog**: [TODO]\n\n## [1.21.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.21.0) (2025-04-17)\n\n#### New experimental features\n\n#### Bug fixes\n\n#### Other notable changes\n- [#847](https://github.com/timescale/timescaledb-toolkit/pull/847): Added `total` accessor for tdigest and uddsketch\n- [#853](https://github.com/timescale/timescaledb-toolkit/pull/853): Performance improvements for `UDDSketch`\n\n#### Shout-outs\n\n**Full Changelog**: [TODO]\n\n## [1.19.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.19.0) (2024-11-14)\n\n#### New experimental features\n\n#### Bug fixes\n\n#### Other notable changes\n\n#### Shout-outs\n\n**Full Changelog**: [TODO]\n\n## [1.18.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.18.0) (2023-11-28)\n\n#### New experimental features\n\n- [#776](https://github.com/timescale/timescaledb-toolkit/pull/776): PostgreSQL 16 support\n\n**Full Changelog**: [TODO]\n\n## [1.17.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.17.0) (2023-07-06)\n\n#### New experimental features\n\n#### Bug fixes\n- [#761](https://github.com/timescale/timescaledb-toolkit/pull/761): Make sure nmost combine uses correct memctx\n\n#### Other notable changes\n\n#### Shout-outs\n\n**Full Changelog**: [TODO]\n\n## [1.16.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.16.0) (2023-04-05)\n\n#### Bug fixes\n- [#733](https://github.com/timescale/timescaledb-toolkit/pull/733): Fix a bug when rolling up overlapping heartbeat_aggs\n- [#740](https://github.com/timescale/timescaledb-toolkit/pull/740): When interpolating an 'locf' time weighted average, extend last point to interpolation boundary\n- [#742](https://github.com/timescale/timescaledb-toolkit/pull/742): Ignore incoming NULL values in hyperloglog rollup\n\n#### Stabilized features\n- [#741](https://github.com/timescale/timescaledb-toolkit/pull/741): Stabilize `approx_count_distinct`\n- [#748](https://github.com/timescale/timescaledb-toolkit/pull/748): Stabilize `approx_percentile_array`\n- [#745](https://github.com/timescale/timescaledb-toolkit/pull/745): Stabilize date utility functions\n- [#751](https://github.com/timescale/timescaledb-toolkit/pull/751): Stabilize `min_n`/`max_n`/`min_n_by`/`max_n_by`\n- [#752](https://github.com/timescale/timescaledb-toolkit/pull/752): Stabilize `mcv_agg`, this was previously our `topn_agg`\n\n#### Other notable changes\n- [#743](https://github.com/timescale/timescaledb-toolkit/pull/743): Remove support for direct upgrades from toolkit versions more than 1 year old. Toolkit versions 1.4.x and 1.5.x will have to upgrade to an intermediate version before upgrading to 1.16.0.\n- [#744](https://github.com/timescale/timescaledb-toolkit/pull/744): Fix nightly CI failures from building TimescaleDB on Enterprise Linux\n- [#749](https://github.com/timescale/timescaledb-toolkit/pull/749): Added num_live_ranges, num_gaps, and trim_to accessors for heartbeat aggregates\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.15.0...1.16.0\n\n## [1.15.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.15.0) (2023-03-08)\n\n#### New experimental features\n\n#### Bug fixes\n- [#715](https://github.com/timescale/timescaledb-toolkit/pull/715): Fix out-of-bounds indexing error in `state_agg` rollup\n\n#### Stabilized features\n- [#722](https://github.com/timescale/timescaledb-toolkit/pull/722): Stabilize heartbeat aggregate.\n- [#724](https://github.com/timescale/timescaledb-toolkit/pull/724): Stabilize integral and interpolated_integral for time-weighted-average.\n- [#723](https://github.com/timescale/timescaledb-toolkit/pull/723): Stabilized `state_agg`\n\n#### Other notable changes\n- [#716](https://github.com/timescale/timescaledb-toolkit/issues/716): Add arrow operator support for counter aggregate and time-weighted aggregate interpolated accessors.\n- [#716](https://github.com/timescale/timescaledb-toolkit/issues/716): Remove experimental versions of interpolated accessors for counter aggregate and time-weighted aggregates.  The stable versions introduced in 1.14.0 should be used instead.\n- [#723](https://github.com/timescale/timescaledb-toolkit/pull/723): Added `state_at` function for `state_agg`\n- [#709](https://github.com/timescale/timescaledb-toolkit/pull/709): Updated pgx version to 0.7.1\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.14.0...1.15.0\n\n## [1.14.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.14.0) (2023-02-09)\n\n#### New experimental features\n\n#### Bug fixes\n- [#660](https://github.com/timescale/timescaledb-toolkit/issues/660): Heartbeat aggregate rollup should interpolate aggregates\n- [#679](https://github.com/timescale/timescaledb-toolkit/issues/679): Heartbeat agg rollup producing invalid aggregates.\n\n#### Stabilized features\n- [#701](https://github.com/timescale/timescaledb-toolkit/pull/701): Stabilize candlestick.\n- [#650](https://github.com/timescale/timescaledb-toolkit/pull/650): Stabilize interpolated_delta & interpolated_rate for counter aggregate, and interpolated_average for time-weighted aggregate.\n\n#### Other notable changes\n- [#685](https://github.com/timescale/timescaledb-toolkit/issues/685): rollup for freq_agg and topn_agg\n- [#692](https://github.com/timescale/timescaledb-toolkit/pull/692): Support specifying a range to `duration_in` to specify a time range to get states in for state aggregates\n- [#692](https://github.com/timescale/timescaledb-toolkit/pull/692): Removed `next` parameter from interpolated state aggregate functions\n- [#692](https://github.com/timescale/timescaledb-toolkit/pull/692): Renamed `state_agg` to `compact_state_agg` and `timeline_agg` to `state_agg`\n- [#699](https://github.com/timescale/timescaledb-toolkit/pull/699): `interpolated_duration_in`/`duration_in`/`interpolated_state_periods`/`state_periods` have the first two arguments swapped: now the aggregate is first and the state is second\n- [#699](https://github.com/timescale/timescaledb-toolkit/pull/699): `into_values`/`into_int_values` now returns a table with intervals instead of microseconds\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.13.1...1.14.0\n\n## [1.13.1](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.13.1) (2023-01-03)\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.13.0...1.13.1\n\n- [#664](https://github.com/timescale/timescaledb-toolkit/pull/664) Support PostgreSQL 15.\n\n## [1.13.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.13.0) (2022-12-13)\n\n#### New experimental features\n- [#615](https://github.com/timescale/timescaledb-toolkit/pull/615): Heartbeat aggregate\n\n  Users can use the new `heartbeat_agg(timestamp, start_time, agg_interval, heartbeat_interval)` to track the liveness of a system in the range (`start_time`, `start_time` + `agg_interval`). Each timestamp seen in that range is assumed to indicate system liveness for the following `heartbeat_interval`.\n\n  Once constructed, users can query heartbeat aggregates for `uptime` and `downtime`, as well as query for `live_ranges` or `dead_ranges`. Users can also check for `live_at(timestamp)`.\n\n  Heartbeat aggregates can also interpolated to better see behavior around the boundaries of the individual aggregates.\n\n- [#620](https://github.com/timescale/timescaledb-toolkit/pull/620): Expose TDigest type\n\n  This is a prototype for building `TDigest` objects client-side, for `INSERT` into tables.\n\n  This is a lightly tested prototype; try it out at your own risk!\n\n  [Examples](docs/examples/)\n\n- [#635](https://github.com/timescale/timescaledb-toolkit/pull/635): AsOf joins for timevectors\n\n  This allows users to join two timevectors with the following semantics `timevectorA -> asof(timevectorB)`. This will return records with the LOCF value from timevectorA at the timestamps from timevectorB. Specifically the returned records contain, for each value in timevectorB, {the LOCF value from timevectorA, the value from timevectorB, the timestamp from timevectorB}.\n\n- [#609](https://github.com/timescale/timescaledb-toolkit/pull/609): New `approx_percentile_array()` function\n\n  Users can use the new `toolkit_experimental.approx_percentile_array(percentiles)` to generate an array of percentile results instead of having to call and rebuild the aggregate multiple times.\n\n- [#636](https://github.com/timescale/timescaledb-toolkit/pull/636): New `timeline_agg` aggregate, which is similar to `state_agg` but tracks the entire state timeline instead of just the duration in each state.\n\n- [#640](https://github.com/timescale/timescaledb-toolkit/pull/640): Support `rollup` for `state_agg` and `timeline_agg`.\n- [#640](https://github.com/timescale/timescaledb-toolkit/pull/640): Support integer states for `state_agg` and `timeline_agg`.\n\n- [#638](https://github.com/timescale/timescaledb-toolkit/pull/638): Introducing Time Vector Templates.\n\nUsers can use the new experimental function `toolkit_experimental.to_text(timevector(time, value),format_string)` to render a formatted text representation of their time vector series. These changes also include `toolkit_experimental.to_plotly(timevector(time, value))`, which will render your time vector series in a format suitable for use with plotly.\n\n#### Bug fixes\n- [#644](https://github.com/timescale/timescaledb-toolkit/pull/644): Fix bug in Candlestick aggregate and reenable partial aggregation.\n\n#### Other notable changes\n- [#646](https://github.com/timescale/timescaledb-toolkit/pull/646): Added experimental support for PostgreSQL 15.\n- [#621](https://github.com/timescale/timescaledb-toolkit/pull/621): Rocky Linux 9 support\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.12.1...1.13.0\n\n## [1.12.1](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.12.1) (2022-11-17)\n\n#### Bug fixes\n- [#624](https://github.com/timescale/timescaledb-toolkit/pull/624): Remove partial aggregation for Candlestick aggregates.\n  We've determined that the cause for the bad results lives somewhere in the functions that are used to support partial aggregation.\n  We can at least prevent folks from running the candlestick aggregates in parallel mode and hitting this bug by dropping support for partial aggregation until we've resolved the issue.\n\n## [1.12.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.12.0) (2022-11-08)\n\n#### New experimental features\n- [#596](https://github.com/timescale/timescaledb-toolkit/pull/596): Introduce Candlestick Aggregate.\n  Users can use either the `toolkit_experimental.candlestick_agg(timestamp, price, volume)` aggregate or the `toolkit_experimental.candlestick(timestamp, open, high, low, close, volume)` function, depending on whether they are starting from tick data or already aggregated data.\n  Both the aggregate form and the function form of `Candlestick` support the following (experimental) accessors (in addition to being re-aggregated via `rollup`):\n  `open`, `high`, `low`, `close`, `open_time`, `high_time`, `low_time`, `close_time`, `volume`, `vwap` (Volume Weighted Average Price)\n  *NOTE*: This functionality improves upon and replaces the need for `toolkit_experimental.ohlc` which will be removed in the next release.\n\n- [#590](https://github.com/timescale/timescaledb-toolkit/pull/590): New `min_n`/`max_n` functions and related `min_n_by`/`max_n_by`.\n  The former is used to get the top N values from a column while the later will also track some additional data, such as another column or even the entire row.\n  These should give the same results as a `SELECT ... ORDER BY ... LIMIT n`, except they can be composed and combined like other toolkit aggregates.\n\n#### Bug fixes\n\n- [#568](https://github.com/timescale/timescaledb-toolkit/pull/568): Allow `approx_count` accessor function to take NULL inputs.\n- [#574](https://github.com/timescale/timescaledb-toolkit/pull/574): Add default unit to interpolated_integral.\n\n#### Other notable changes\n\n- RPM packages for CentOS 7 have returned.\n- New Homebrew formula available for macOS installation: `brew install timescale/tap/timescaledb-toolkit`.\n- [#547](https://github.com/timescale/timescaledb-toolkit/pull/547): Update pgx to 0.5.0. This is necessary for adding Postgres 15 support coming soon.\n- [#571](https://github.com/timescale/timescaledb-toolkit/pull/571): Update CI docker image for pgx 0.5.0.\n- [#599](https://github.com/timescale/timescaledb-toolkit/pull/599): Reduce floating point error when using `stats_agg` in moving aggregate mode.\n- [#589](https://github.com/timescale/timescaledb-toolkit/pull/589): Update pgx to 0.5.4.\n- [#594](https://github.com/timescale/timescaledb-toolkit/pull/594): Verify that pgx doesn't generate CREATE OR REPLACE FUNCTION.\n- [#592](https://github.com/timescale/timescaledb-toolkit/pull/592): Add build script option to install in release mode.\n\n#### Shout-outs\n\n- @zyro for reporting null handling issue on `count_min_sketch`.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.11.0...1.12.0\n\n## [1.11.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.11.0) (2022-09-21)\n\n#### New experimental features\n\n- arm64/aarch64 DEB packages are now available for Ubuntu 20.04 (focal) & 22.04 (jammy), and Debian 10 (buster) & 11 (bulleye).\n- [#526](https://github.com/timescale/timescaledb-toolkit/pull/526): Add `integral` and `interpolated_integral` functions for the time_weight aggregate. Makes `trapezoidal` an alias for `linear` in `time_weight` as it might be a more familiar numeric integral method for some.\n- [#517](https://github.com/timescale/timescaledb-toolkit/pull/517): Add a gap preserving `lttb` named `gp_lttb` to handle downsampling of data with large gaps.\n- [#513](https://github.com/timescale/timescaledb-toolkit/pull/513): Add `first_val`, `last_val`, `first_time` and `last_time` to `time_weight` and `counter_agg` to access the first and the last data points within the aggregate data structures.\n- [#527](https://github.com/timescale/timescaledb-toolkit/pull/527): Rename `{open, high, low, close}_at` to `{open, high, low, close}_time` to be consistent with newly added `first_time` and `last_time` accessor functions.\n\n#### Stabilized features\n\n- [#498](https://github.com/timescale/timescaledb-toolkit/pull/498): Stabilize `asap_smooth` aggregate.\n\n#### Bug fixes\n\n- [#509](https://github.com/timescale/timescaledb-toolkit/pull/509), [#531](https://github.com/timescale/timescaledb-toolkit/pull/531): Fix bugs in`hyperloglog`. Error rates are now significantly more consistent when the number of buckets are close to the actual cardinality.\n- [#514](https://github.com/timescale/timescaledb-toolkit/pull/514): Fix a bug in `toolkit_experimental.interpolated_delta`.\n- [#503](https://github.com/timescale/timescaledb-toolkit/pull/503): Fix bitwise logic in timevector combine.\n- [#507](https://github.com/timescale/timescaledb-toolkit/pull/507): Fix a typo in `approx_count_distinct`.\n\n#### Other notable changes\n\n- DEB packages for Ubuntu 18.04 (Bionic) on amd64 are now available.\n- [#536](https://github.com/timescale/timescaledb-toolkit/pull/536): Document equirement to use same compiler for cargo-pgx and Toolkit.\n- [#535](https://github.com/timescale/timescaledb-toolkit/pull/535): Make tests pass in Canadian locales. \n- [#537](https://github.com/timescale/timescaledb-toolkit/pull/537): Enforce `cargo fmt` in CI.\n- [#524](https://github.com/timescale/timescaledb-toolkit/pull/524): Updating Toolkit To Start Using Cargo Fmt.\n- [#522](https://github.com/timescale/timescaledb-toolkit/pull/522): Move update-tester tests to markdown files.\n\n#### Shout-outs\n\n- @BenSandeen for fixing typos and errors in the hyperloglog++ implementation.\n- @jaskij for reporting security advisories and suggestion on documenting support for PG 14.\n- @jeremyhaberman for fixing a typo in `APPROX_COUNT_DISTINCT_DEFAULT_SIZE`.\n- @jledentu for reporting an error on `interpolated_delta`.\n- @stevedrip for a very detailed bug report on hyperloglog++ and suggestions for fixing it.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.10.1...1.11.0\n\n## [1.10.1](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.10.1) (2022-08-18)\n\n#### New experimental features\n\n- [#490](https://github.com/timescale/timescaledb-toolkit/pull/490): Month normalization function `month_normalize` and the helper function `days_in_month`, useful for normalizing data to a fixed month length for more meaningful month-to-month comparison.\n- [#496](https://github.com/timescale/timescaledb-toolkit/pull/496): `OHLC` aggregate, and the associated `rollup` and accessor functions `open`, `high`, `low`, `close`, `{open, high, low, close}_at` mainly for trading data.\n\n#### Stabilized features\n\n- [#495](https://github.com/timescale/timescaledb-toolkit/pull/495): `LTTB` downsampling function.\n- [#491](https://github.com/timescale/timescaledb-toolkit/pull/491), [#488](https://github.com/timescale/timescaledb-toolkit/pull/488): The arrow operators (->) of the accessor functions for `stats_agg`, `percentile_agg`, `counter_agg`, `gauge_agg` and `hyperloglog`. As an example, `average` accessor can now be used with `stats_agg` like this,\n    ```SQL\n    select location, \n        stats_agg(temperature) -> average() AS avg_temperature\n    from conditions \n    group by location\n    ```\n\n#### Bug fixes\n\n- [#465](https://github.com/timescale/timescaledb-toolkit/pull/465): Off by one error in state_agg interpolate.\n\n#### Other notable changes\n\n- Fix an issue where the 1.9.0 release unintentionally identified the toolkit extension version as 1.10.0-dev in the postgresql control file.\n- [#467](https://github.com/timescale/timescaledb-toolkit/pull/467): Document supported platforms in Readme.\n- [#463](https://github.com/timescale/timescaledb-toolkit/pull/463): Use pg14 as an example for instructions in  instead of pg13. Add reference to deb and rpm packages.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.8.0...1.10.1\n\n## [1.9.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.9.0) (2022-08-16)\n\n**An incorrect version (1.10.0-dev) was used which can cause upgrade failures. Not made GA.**\n\n## [1.8.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.8.0) (2022-07-05)\n\n#### New experimental features\n\n- [#454](https://github.com/timescale/timescaledb-toolkit/pull/454): Saturating Math for i32/integers:\n    - `saturating_add`\n    - `saturating_add_pos`\n    - `saturating_sub`\n    - `saturating_sub_pos`\n    - `saturating_mul`\n- [#456](https://github.com/timescale/timescaledb-toolkit/pull/456): Adding interpolating accessors:\n    - `interpolated_duration_in` to `state_agg`, \n    - `interpolated_average` to `time_weight`, `interpolated_delta`\n    - `interpolated_rate` to `counter_agg` and `gauge_agg`.\n- [#388](https://github.com/timescale/timescaledb-toolkit/pull/388): Create Count-Min Sketch crate.\n- [#459](https://github.com/timescale/timescaledb-toolkit/pull/459): Add a convenient `approx_count_distinct` function which internally uses hyperloglog with a default bucket size of 2^15.\n- [#458](https://github.com/timescale/timescaledb-toolkit/pull/458): Add `count_min_sketch` aggregate and `approx_count` accessor.\n- [#434](https://github.com/timescale/timescaledb-toolkit/pull/434): Initial changes to support aarch64-unknown-linux-gnu.\n\n#### Bug fixes\n\n- [#429](https://github.com/timescale/timescaledb-toolkit/pull/429): Support explicit NULL values in timevectors.\n- [#441](https://github.com/timescale/timescaledb-toolkit/pull/441): Relax tolerance in UDDSketch merge assertions.\n- [#444](https://github.com/timescale/timescaledb-toolkit/pull/444): Fix default collation deserialization.\n\n#### Other notable changes\n\n- [#451](https://github.com/timescale/timescaledb-toolkit/pull/451): Improve error message for HyperLogLog.\n- [#417](https://github.com/timescale/timescaledb-toolkit/pull/417): Include both pgx 0.2.x and pgx 0.4.x in CI image.\n- [#416](https://github.com/timescale/timescaledb-toolkit/pull/416): Prepare for the 1.8.0 cycle.\n- [#418](https://github.com/timescale/timescaledb-toolkit/pull/418): Made update-tester require two versions of cargo-pgx.\n- [#421](https://github.com/timescale/timescaledb-toolkit/pull/421): Don't install pgx as root or under \"/\".\n- [#427](https://github.com/timescale/timescaledb-toolkit/pull/427): Fix failing update-tester in CI.\n- [#428](https://github.com/timescale/timescaledb-toolkit/pull/428): Update github cache keys.\n- [#430](https://github.com/timescale/timescaledb-toolkit/pull/430): Lock pgx versions all the way.\n- [#408](https://github.com/timescale/timescaledb-toolkit/pull/408): Upgrade to pgx 0.4.5.\n- [#436](https://github.com/timescale/timescaledb-toolkit/pull/436): Change which cargo-pgx subcommand is added to PATH in CI image.\n- [#432](https://github.com/timescale/timescaledb-toolkit/pull/432): Remove PATH hack in tools/build script.\n- [#437](https://github.com/timescale/timescaledb-toolkit/pull/437): GitHub Actions improvements.\n- [#448](https://github.com/timescale/timescaledb-toolkit/pull/448): Run clippy GitHub Actions job without qualification.\n- [#446](https://github.com/timescale/timescaledb-toolkit/pull/446): Update README.md.\n- [#414](https://github.com/timescale/timescaledb-toolkit/pull/414): Specify Ubuntu 20.04 instead of 'latest' in github configuration.\n\n#### Shout-outs\n\n- @tyhoff for reporting UDDSketch assertion error [#396](https://github.com/timescale/timescaledb-toolkit/issues/396).\n- @hardikm10 for reporting hyperloglog deserialization issue [#443](https://github.com/timescale/timescaledb-toolkit/issues/443).\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.7.0...1.8.0\n\n## [1.7.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.7.0) (2022-05-10)\n\n#### New experimental features\n\n- [#389](https://github.com/timescale/timescaledb-toolkit/pull/389): Create typed specialization for `freq_agg` and `topn_agg`.\n\n#### Bug fixes\n\n- [#378](https://github.com/timescale/timescaledb-toolkit/pull/378): Return INTERVAL from `duration_in(TEXT, StateAgg)` instead of `i64`.\n- [#379](https://github.com/timescale/timescaledb-toolkit/pull/379): Handle NULL output from our aggregates: `asap`, `counter_agg`, `freq_agg`, `gauge_agg`, `hyperloglog`, `lttb`, `stats_agg`, `tdigest`, `uddsketch`.\n\n#### Other notable changes\n- [#367](https://github.com/timescale/timescaledb-toolkit/pull/367): Switch stabilization tests to new info, meaning that there's one central location for stabilization info.\n- [#372](https://github.com/timescale/timescaledb-toolkit/pull/372): Improve tools/build flexibility for local builds.\n- [#394](https://github.com/timescale/timescaledb-toolkit/pull/394): Copy almost all the counter_agg functions for gauge_agg.\n- [#395](https://github.com/timescale/timescaledb-toolkit/pull/395): Remove GUC as they are no longer needed.\n- [#399](https://github.com/timescale/timescaledb-toolkit/pull/399): Allow manual packaging.\n- [#405](https://github.com/timescale/timescaledb-toolkit/pull/405): Update CI to rust 1.60.\n- [#407](https://github.com/timescale/timescaledb-toolkit/pull/407): Update postgres versions in ci Dockerfile.\n- [#409](https://github.com/timescale/timescaledb-toolkit/pull/409): Make dependencies version explicit in our CI image.\n- [#404](https://github.com/timescale/timescaledb-toolkit/pull/404): Refactor TimeVector to greatly simplify structure.\n- [#412](https://github.com/timescale/timescaledb-toolkit/pull/412): Allow building CI image in Actions.\n- [#411](https://github.com/timescale/timescaledb-toolkit/pull/411), [#413](https://github.com/timescale/timescaledb-toolkit/pull/413): Create reportpackagingfailures.yml for reporting packaging failures not from CI builds.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.6.0...1.7.0\n\n## [1.6.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.6.0) (2022-03-29)\n\n#### New experimental features\n\n- [#330](https://github.com/timescale/timescaledb-toolkit/pull/330): Add serialization for FrequencyTransState.\n- [#368](https://github.com/timescale/timescaledb-toolkit/pull/368): Add `into_values` function for `state_agg`.\n- [#370](https://github.com/timescale/timescaledb-toolkit/pull/370): Add a `topn (topn_agg)` variant of `freq_agg`, which is more convenient to use.\n- [#375](https://github.com/timescale/timescaledb-toolkit/pull/375): Add `gauge_agg` and associated accessor functions `delta`, `idelta_left`, `idelta_right`, and the `rollup` function.\n\n#### Other notable changes\n\n- [#332](https://github.com/timescale/timescaledb-toolkit/pull/332): Speed up builds by fixing github action cache and cargo build cache.\n- [#377](https://github.com/timescale/timescaledb-toolkit/pull/377): Stop auto building _nightly_ image.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.5.2...1.6.0\n\n## [1.5.2](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.5.2) (2022-03-07)\n\n**HIGH PRIORITY SECURITY UPDATE**.\n\n#### Bug fixes\n\n- There's a vulnerability in Toolkit 1.5 and earlier due to the fact that it creates a PLPGSQL function using CREATE OR REPLACE and without properly locking down the search path. This means that a user could pre-create the trigger function to run arbitrary code. To fix this we remove the trigger entirely; it no longer pulls its weight. This fix locks down our update scripts to only use CREATE OR REPLACE when actually necessary; while we don't yet have an exploit for the other functions, it would be unsurprising if one exists.\n- [#351](https://github.com/timescale/timescaledb-toolkit/pull/351): Make serialize functions strict to handle NULL values in partitioned aggregates.\n\n#### Shout-outs\n\n- @svenklemm for reporting the vulnerability.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.5.0...1.5.2\n\n## [1.5.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.5.0) (2022-01-31)\n\n**The first version that unifies the community build with Timescale Cloud build.**\n\n#### New experimental features\n\n- `freq_agg` for estimating the most common elements in a column.\n- `state_agg` for measuring the total time spent in different states.\n\n#### Other notable changes\n\n- Enforce clippy linting.\n- Update rust to 1.57.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.4.0...1.5.0\n\n## [1.4.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.4.0), [1.4.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.4.0-cloud) (2021-11-17)\n\n#### Stabilized features\n\n- Postgres 14 support.\n\n#### Other notable changes\n\n- Upgrade pgx to 0.2.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.3.1...1.4.0-cloud\n\n## [1.3.1](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.3.1), [1.3.1-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.3.1-cloud) (2021-10-27)\n\n#### Stabilized features\n\n- Postgres 14 support.\n\n#### Other notable changes\n\n- Upgrade pgx to 0.2.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.3.0...1.3.1-cloud\n\n## [1.3.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.3.0), [1.3.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.3.0-cloud) (2021-10-18)\n\n#### New experimental features\n\n- `timevector` function pipelines: a compact and more readable way to perform a sequence of analytic operations such as the following one,\n    ```\n    timevector(ts, val) -> sort() -> delta() -> abs() -> sum()\n    ```\n- `->` accessor for Toolkit types enables syntax like `stats_agg(data) -> average()`.\n- `to_epoch()` wrapper for `extract ('EPOCH' FROM timestamp)` that makes it work more like an inverse of `to_timestamp(DOUBLE PRECISION)`.\n#### Stabilized features\n\n- `counter_agg` helper functions for Prometheus-style resetting monotonic counters.\n- `hyperloglog` efficient approximate COUNT DISTINCT.\n- `stats_agg` two-step aggregate for common statistics.\n\n#### Other notable changes\n\n- This release changes the textual I/O format for Toolkit types. We are uncertain if we will need to do so again in the future. Due to this we currently only support dump/restore within a single version of the extension.\n\n#### Shout-outs\n\n- @jonatas for the contribution [#237](https://github.com/timescale/timescaledb-toolkit/pull/237).\n- @burmecia for the contribution [#251](https://github.com/timescale/timescaledb-toolkit/pull/251).\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.2.0...1.3.0-cloud\n\n## [1.2.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.2.0), [1.2.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.2.0-cloud) (2021-09-14)\n\n#### New experimental features\n\n- Refinements to `hyperloglog` including a function to report relative error and fixing the functionality of `rollup`.\n- Introduction of a `topn` approximation API. Presently this will only work for integer data, but expect to see further refinements that greatly expand this behavior.\n- New `map_series` and `map_data` pipeline elements for the time series API that allow uses to provide custom transforms of their time series data. Additionally introduced a `|>>` pipeline operator for an even more streamlined interface into the new mapping functionality.\n\n#### Bug fixes\n\n- Make a pass through all toolkit functions to correctly label behavior as immutable and parallel safe. This should improve the optimizations Postgres can apply to toolkit plans, particularly when run in a Timescale multinode cluster.\n- Improve handling of internal data structures to reduce extraneous copies of data.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.1.0...1.2.0-cloud\n\n## [1.1.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.1.0), [1.1.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.1.0-cloud) (2021-08-04)\n\n#### New experimental features\n\n- `hyperloglog` has been updated to use Hyperloglog++ under the hood. This does not change the user-facing API but should improve the accuracy of hyperloglog() estimates. This is the last major change expected for hyperloglog() and is now a candidate for stabilization pending user feedback.\n- We've started experimenting with the pipeline API. While it's still very much a work in progress, it's at a point where the high-level concepts should be understandable. For example, a pipeline that outputs the daily change of a set of data, interpolating away any gaps in daily data, could look like\n    ```\n    SELECT timeseries(time, val)\n        |> sort()\n        |> resample_to_rate('trailing_average', '24 hours', true)\n        |> fill_holes('interpolate')\n        |> delta()\n    FROM ...\n    ```\n    It's still early days for this API and it is not yet polished, but we would love feedback about its direction.\n\n#### Bug fixes\n\n- Fix a small memory leak in aggregation functions. This could have leaked ≈8 bytes per aggregate call.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/1.0.0...1.1.0-cloud\n\n## [1.0.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.0.0), [1.0.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/1.0.0-cloud) (2021-07-12)\n\n**This release renames the extension to `TimescaleDB Toolkit` from Timescale Analytics and starts stabilizing functionality.**\n\n#### New experimental features\n\n- `stats_agg()` eases the analysis of more sophisticated bucketed statistics, such as rolling averages. (Docs are forthcoming, until then fell free to peruse the design discussion doc.\n- `timeseries` which will serve as a building block for many pipelines, and unifies the output of lttb and ASAP.\n\n#### Stabilized features\n\n- Percentile-approximation algorithms including `percentile_agg()`, `uddsketch()` and `tdigest()` along with their associated functions. These are especially useful for computing percentiles in continuous aggregates.\n- [Time-weighted average](https://github.com/timescale/timescaledb-toolkit/blob/main/docs/time_weighted_average.md) along with its associated functions. This eases taking the average over an irregularly spaced dataset that only includes changepoints.\n\n#### Other notable changes\n\n- The on-disk layout `uddsketch` has be reworked to store buckets compressed. This can result in an orders-of-magnitude reduction in it's storage requirements.\n- The textual format `uddsketch` has been reworked to be more readable.\n- Functions that take in a `uddsketch` or `tdigest` have been reworked to be 0-copy when applicable, improving the performance of such functions by 10-100x.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/0.3.0...1.0.0-cloud\n\n## [0.3.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/0.3.0), [0.3.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/0.3.0-cloud) (2021-06-17)\n\n#### Other notable changes\n\n- Internal improvements.\n- Largely prep work for the upcoming 1.0 release.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/0.2.0...0.3.0-cloud\n\n## [0.2.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/0.2.0) (2021-04-08), [0.2.0-cloud](https://github.com/timescale/timescaledb-toolkit/releases/tag/0.2.0-cloud) (2021-04-29)\n\n#### New experimental features\n\n- ASAP Smoothing (`asap_smooth`) – A graph smoothing algorithm that highlights changes.\n- Counter Aggregates (`counter_agg`) – Tools to ease working with reset-able counters.\n- Largest Triangle Three Buckets (`lttb`) – A downsampling algorithm that tries to preserve visual similarity.\n- Time Bucket Range – A version of `time_bucket()` that outputs the [start, end) times of the bucket.\n- Update `UddSketch` with an aggregate that merges multiple `UddSketchs` and various internal improvements.\n\n**Full Changelog**: https://github.com/timescale/timescaledb-toolkit/compare/0.1.0...0.2.0-cloud\n\n## [0.1.0](https://github.com/timescale/timescaledb-toolkit/releases/tag/0.1.0) (2021-03-03)\n\n#### New experimental features\n\n- `hyperloglog` – An approximate COUNT DISTINCT based on hashing that provides reasonable accuracy in constant space.\n- `tdigest` – A quantile estimate sketch optimized to provide more accurate estimates near the tails (i.e. 0.001 or 0.995) than conventional approaches.\n- `uddsketch` – A quantile estimate sketch which provides a guaranteed maximum relative error.\n- Time-weighted average (`time_weight`) – A time-weighted averaging function to determine the value of things proportionate to the time they are set.\n\n#### Stabilized features\n\n- None. All features are experimental.\n"
  },
  {
    "path": "LICENSE",
    "content": "Unless otherwise Source code in this repository, and any binaries built from\nthis source code, in whole or in part, are licensed under the\nTimescale License (the \"License\"). You may not use these files except in\ncompliance with the License.\n\nYou may obtain a copy of the License at\n\n   https://github.com/timescale/timescaledb/blob/master/tsl/LICENSE-TIMESCALE\n"
  },
  {
    "path": "NOTICE",
    "content": "TimescaleDB-Toolkit (TM)\n\nCopyright (c) 2021-2024  Timescale, Inc. All Rights Reserved.\n\nUnless otherwise stated, source code in this repository, and any binaries built\nfrom this source code, in whole or in part, are licensed under the\nTimescale License (the \"License\"). You may not use these files except in\ncompliance with the License.\n\nYou may obtain a copy of the License at\n\n   https://github.com/timescale/timescaledb/blob/master/tsl/LICENSE-TIMESCALE\n"
  },
  {
    "path": "Readme.md",
    "content": "[![CI](https://github.com/timescale/timescaledb-toolkit/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/timescale/timescaledb-toolkit/actions/workflows/ci.yml)\n\n# TimescaleDB Toolkit\n\nThis repository is the home of the TimescaleDB Toolkit team. Our mission is to\nease all things analytics when using TimescaleDB, with a particular focus on\ndeveloper ergonomics and performance. Our issue tracker contains more\non [the features we're planning to work on](https://github.com/timescale/timescaledb-toolkit/labels/proposed-feature)\nand [the problems we're trying to solve](https://github.com/timescale/timescaledb-toolkit/labels/feature-request).\n\nDocumentation for this version of the TimescaleDB Toolkit extension can be found\nin this repository at [`docs`](https://github.com/timescale/timescaledb-toolkit/tree/main/docs).\nThe release history can be found on this repo's [GitHub releases](https://github.com/timescale/timescaledb-toolkit/releases).\n\n## 🖥 Try It Out\n\nThe extension comes pre-installed on all [Tiger Cloud](https://www.tigerdata.com/cloud) instances and also on our full-featured [`timescale/timescaledb-ha` docker image](https://hub.docker.com/r/timescale/timescaledb-ha).\n\nIf DEB and RPM packages are a better fit for your situation, refer to the [Install Toolkit on self-hosted TimescaleDB](https://docs.timescale.com/self-hosted/latest/tooling/install-toolkit/#install-toolkit-on-self-hosted-timescaledb) how-to guide for further instructions on installing the extension via your package manager.\n\nAll versions of the extension contain experimental features in the `toolkit_experimental` schema. See [our docs section on experimental features](/docs/README.md#tag-notes) for more details.\n\n## 💿 Installing From Source\n\n### Supported platforms\n\nThe engineering team regularly tests the extension on the following platforms:\n\n- x86_64-unknown-linux-gnu (Ubuntu Linux 24.04) (tested prior to every merge)\n- aarch64-unknown-linux-gnu (Ubuntu Linux 24.04) (tested at release time)\n- x86_64-apple-darwin (MacOS 12) (tested frequently on eng workstation)\n- aarch64-apple-darwin (MacOS 12) (tested frequently on eng workstation)\n\nAs for other platforms: patches welcome!\n\n### 🔧 Tools Setup\n\nBuilding the extension requires valid [rust](https://www.rust-lang.org/) (we build and test on 1.65), [rustfmt](https://github.com/rust-lang/rustfmt), and clang installs, along with the postgres headers for whichever version of postgres you are running, and pgrx.\nWe recommend installing rust using the [official instructions](https://www.rust-lang.org/tools/install):\n\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n```\n\nand build tools, the postgres headers, in the preferred manner for your system. You may also need to install OpenSSl.\nFor Ubuntu you can follow the [postgres install instructions](https://www.postgresql.org/download/linux/ubuntu/) then run\n\n```bash\nsudo apt-get install make gcc pkg-config clang postgresql-server-dev-14 libssl-dev\n```\n\nNext you need [cargo-pgrx](https://github.com/tcdi/pgrx), which can be installed with\n\n```bash\ncargo install --version '=0.16.1' --force cargo-pgrx\n```\n\nYou must reinstall cargo-pgrx whenever you update your Rust compiler, since cargo-pgrx needs to be built with the same compiler as Toolkit.\n\nFinally, setup the pgrx development environment with\n\n```bash\ncargo pgrx init --pg14 pg_config\n```\n\nInstalling from source is also available on macOS and requires the same set of prerequisites and set up commands listed above.\n\n### 💾 Building and Installing the extension\n\nDownload or clone this repository, and switch to the `extension` subdirectory, e.g.\n\n```bash\ngit clone https://github.com/timescale/timescaledb-toolkit && \\\ncd timescaledb-toolkit/extension\n```\n\nThen run\n\n```\ncargo pgrx install --release && \\\ncargo run --manifest-path ../tools/post-install/Cargo.toml -- pg_config\n```\n\nTo initialize the extension after installation, enter the following into `psql`:\n\n```\nCREATE EXTENSION timescaledb_toolkit;\n```\n\n## ✏️ Get Involved\n\nWe appreciate your help in shaping the project's direction! Have a look at the\n[list of features we're thinking of working on](https://github.com/timescale/timescaledb-toolkit/labels/proposed-feature)\nand feel free to comment on the features or expand the list.\n\n### 🔨 Testing\n\nSee above for prerequisites and installation instructions.\n\nYou can run tests against a postgres version `pg15`, `pg16`, `pg17`, or `pg18` using\n\n```\ncargo pgrx test ${postgres_version}\n```\n\n## Learn about Tiger Data\n\nTiger Data is the fastest PostgreSQL for transactional, analytical, and agentic workloads. To learn more about the company and its products, visit [tigerdata.com](https://www.tigerdata.com)."
  },
  {
    "path": "crates/aggregate_builder/Cargo.toml",
    "content": "[package]\nname = \"aggregate_builder\"\nversion = \"0.1.0\"\nedition = \"2018\"\n\n[lib]\nproc-macro = true\n\n[dependencies]\nsyn = {version=\"1.0\", features=[\"extra-traits\", \"visit\", \"visit-mut\", \"full\"]}\nquote = \"1.0\"\nproc-macro2 = \"1.0\"\n\n[features]\nprint-generated = []\n"
  },
  {
    "path": "crates/aggregate_builder/Readme.md",
    "content": "# Aggregate Builder #\n\nLibrary for building Postgres [aggregate functions](https://www.postgresql.org/docs/current/xaggr.html)\nthat imitates\n[`CREATE AGGREGATE`](https://www.postgresql.org/docs/current/sql-createaggregate.html).\n\n## Syntax ##\n\nCurrent syntax looks something like like\n\n```rust\n#[aggregate] impl aggregate_name {\n    type State = InternalTransitionType;\n\n    fn transition(\n        state: Option<State>,\n        #[sql_type(\"sql_type\")] argument: RustType, // can have an arbitrary number of args\n    ) -> Option<State> {\n        // transition function function body goes here\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<ResultType> {\n        // final function function body goes here\n    }\n\n    // the remaining items are optional\n\n    // parallel-safety marker if desirable\n    const PARALLEL_SAFE: bool = true;\n\n    fn serialize(state: &State) -> bytea {\n        // serialize function body goes here\n    }\n\n    fn deserialize(bytes: bytea) -> State {\n        // deserialize function body goes here\n    }\n\n    fn combine(state1: Option<&State>, state2: Option<&State>) -> Option<State> {\n        // combine function body goes here\n    }\n}\n```\n\nAll items except for `type State`, `fn transition()`, and `fn finally()` are\noptional. The SQL for the aggregate and its functions will be created\nautomatically, and any necessary memory context switching is handled\nautomatically for most cases¹.\n\n¹It will switch to the aggregate memory context before calling the transition\nfunction body and the combine function body. Looking through `array_agg()`'s\ncode this seems to be the correct places to do so. Note that if you want to\nallocate in the aggregate memory context in the final function other work may\nbe needed.\n\n## Example ##\n\nBelow is a complete example of an `anything()` aggregate that returns one of\nthe aggregated values.\n\n```rust\n#[aggregate] impl anything {\n    type State = String;\n\n    fn transition(\n        state: Option<State>,\n        #[sql_type(\"text\")] value: String,\n    ) -> Option<State> {\n        state.or(Some(value))\n    }\n\n    fn finally(state: Option<&State>) -> Option<String> {\n        state.as_deref().cloned()\n    }\n}\n```\n\n## Expansion ##\n\nIgnoring some supplementary type checking we add to improve error messages, the\nmacro expands aggregate definitions to rust code something like the following\n(explanations as comments in-line)\n```rust\n// we nest things within a module to mimic the namespacing of an `impl` block\npub mod aggregate_name {\n    // glob import to further act like an `impl`\n    use super::*;\n\n    pub type State = String;\n\n    // PARALLEL_SAFE constant in case someone wants to use it\n    // unlikely to be actually used in practice\n    #[allow(dead_code)]\n    pub const PARALLEL_SAFE: bool = true;\n\n    #[pgrx::pg_extern(immutable, parallel_safe)]\n    pub fn aggregate_name_transition_fn_outer(\n        __inner: pgrx::Internal,\n        value: RustType,\n        __fcinfo: pg_sys::FunctionCallInfo,\n    ) -> Option<Internal> {\n        use crate::palloc::{Inner, InternalAsValue, ToInternal};\n        unsafe {\n            // Translate from the SQL type to the rust one\n            // we actually store an `Option<State>` rather than a `State`.\n            let mut __inner: Option<Inner<Option<State>>> = __inner.to_inner();\n            // We steal the state out from under the pointer leaving `None` in\n            // its place. This means that if the inner transition function\n            // panics the inner transition function will free `State` while the\n            // teardown hook in the aggregate memory context will only free inner\n            let inner: Option<State> = match &mut __inner {\n                None => None,\n                Some(inner) => Option::take(&mut **inner),\n            };\n            let state: Option<State> = inner;\n            // Switch to the aggregate memory context. This ensures that the\n            // transition state lives for as long as the aggregate, and that if\n            // we allocate from Postgres within the inner transition function\n            // those too will stay around.\n            crate::aggregate_utils::in_aggregate_context(__fcinfo, || {\n                // call the inner transition function\n                let result = transition(state, value);\n\n                // return the state to postgres, if we have a pointer just store\n                // in that, if not allocate one only if needed.\n                let state: Option<State> = result;\n                __inner = match (__inner, state) {\n                    (None, None) => None,\n                    (None, state @ Some(..)) => Some(state.into()),\n                    (Some(mut inner), state) => {\n                        *inner = state;\n                        Some(inner)\n                    }\n                };\n                __inner.internal()\n            })\n        }\n    }\n    pub fn transition(state: Option<State>, value: String) -> Option<State> {\n        // elided\n    }\n\n    #[pgrx::pg_extern(immutable, parallel_safe)]\n    pub fn aggregate_name_finally_fn_outer(\n        __internal: pgrx::Internal,\n        __fcinfo: pg_sys::FunctionCallInfo,\n    ) -> Option<String> {\n        use crate::palloc::InternalAsValue;\n        unsafe {\n            // Convert to the rust transition type, see the comment in the\n            // transition function for why we store an `Option<State>`\n            let mut input: Option<Inner<Option<State>>> = __internal.to_inner();\n            let input: Option<&mut State> = input.as_deref_mut()\n                .map(|i| i.as_mut())\n                .flatten();\n            // We pass in an `Option<&mut State>`; `Option<>` because the\n            // transition state might not have been initialized yet;\n            // `&mut State` since while the final function has unique access to\n            // the transition function it must leave it a valid state when it's\n            // finished\n            let state: Option<&mut State> = input;\n            finally(state)\n        }\n    }\n    pub fn finally(state: Option<&mut State>) -> Option<String> {\n        // elided\n    }\n\n    #[pgrx::pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\n    pub fn aggregate_name_serialize_fn_outer(__internal: pgrx::Internal) -> bytea {\n        use crate::palloc::{Inner, InternalAsValue};\n        // Convert to the rust transition type, see the comment in the\n        // transition function for why we store an `Option<State>`\n        let input: Option<Inner<Option<State>>> = unsafe { __internal.to_inner() };\n        let mut input: Inner<Option<State>> = input.unwrap();\n        // We pass by-reference for the same reason as the final function.\n        // Note that _technically_ you should not mutate in the serialize,\n        // function though there are cases you can get away with it when using\n        // an `internal` transition type.\n        let input: &mut State = input.as_mut().unwrap();\n        let state: &State = input;\n        serialize(state)\n    }\n    pub fn serialize(state: &State) -> bytea {\n        // elided\n    }\n\n    #[pgrx::pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\n    pub fn aggregate_name_deserialize_fn_outer(\n        bytes: crate::raw::bytea,\n        _internal: Internal,\n    ) -> Option<Internal> {\n        use crate::palloc::ToInternal;\n        let result = deserialize(bytes);\n        let state: State = result;\n        // Convert to the rust transition type, see the comment in the\n        // transition function for why we store an `Option<State>`.\n        // We deliberately don't switch to the aggregate transition context\n        // because the postgres aggregates do not do so.\n        let state: Inner<Option<State>> = Some(state).into();\n        unsafe { Some(state).internal() }\n    }\n    pub fn deserialize(bytes: crate::raw::bytea) -> State {\n        // elided\n    }\n\n    #[pgrx::pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\n    pub fn aggregate_name_combine_fn_outer(\n        a: Internal,\n        b: Internal,\n        __fcinfo: pg_sys::FunctionCallInfo,\n    ) -> Option<Internal> {\n        use crate::palloc::{Inner, InternalAsValue, ToInternal};\n        unsafe {\n            // Switch to the aggregate memory context. This ensures that the\n            // transition state lives for as long as the aggregate, and that if\n            // we allocate from Postgres within the inner transition function\n            // those too will stay around.\n            crate::aggregate_utils::in_aggregate_context(__fcinfo, || {\n                let result = combine(a.to_inner().as_deref(), b.to_inner().as_deref());\n                let state: Option<State> = result;\n                let state = match state {\n                    None => None,\n                    state @ Some(..) => {\n                        let state: Inner<Option<State>> = state.into();\n                        Some(state)\n                    }\n                };\n                state.internal()\n            })\n        }\n    }\n    pub fn combine(a: Option<&State>, b: Option<&State>) -> Option<State> {\n        // elided\n    }\n\n    // SQL generated for the aggregate\n    pgrx::extension_sql!(\"\\n\\\n        CREATE AGGREGATE toolkit_experimental.aggregate_name (value RustType) (\\n\\\n            stype = internal,\\n\\\n            sfunc = toolkit_experimental.aggregate_name_transition_fn_outer,\\n\\\n            finalfunc = toolkit_experimental.aggregate_name_finally_fn_outer,\\n\\\n            parallel = safe,\\n\n            serialfunc = toolkit_experimental.aggregate_name_serialize_fn_outer,\\n\\\n            deserialfunc = toolkit_experimental.aggregate_name_deserialize_fn_outer,\\n\\\n            combinefunc = toolkit_experimental.aggregate_name_combine_fn_outer\\n\\\n        );\\n\",\n        name = \"aggregate_name_extension_sql\",\n        requires = [\n            aggregate_name_transition_fn_outer,\n            aggregate_name_finally_fn_outer,\n            aggregate_name_serialize_fn_outer,\n            aggregate_name_deserialize_fn_outer,\n            aggregate_name_combine_fn_outer,\n        ],\n    );\n}\n```"
  },
  {
    "path": "crates/aggregate_builder/src/lib.rs",
    "content": "use std::borrow::Cow;\n\nuse proc_macro::TokenStream;\n\nuse proc_macro2::{Span, TokenStream as TokenStream2};\n\nuse quote::{quote, quote_spanned};\n\nuse syn::{\n    parse::{Parse, ParseStream},\n    parse_macro_input, parse_quote,\n    punctuated::Punctuated,\n    spanned::Spanned,\n    token::Comma,\n    Token,\n};\n\n#[proc_macro_attribute]\npub fn aggregate(_attr: TokenStream, item: TokenStream) -> TokenStream {\n    // Parse the input tokens into a syntax tree\n    let input = parse_macro_input!(item as Aggregate);\n    let expanded = expand(input);\n    if cfg!(feature = \"print-generated\") {\n        println!(\"{expanded}\");\n    }\n    expanded.into()\n}\n\n//\n// Parser\n//\n\n// like ItemImpl except that we allow `name: Type \"SqlType\"` for `fn transition`\nstruct Aggregate {\n    schema: Option<syn::Ident>,\n    name: syn::Ident,\n\n    state_ty: AggregateTy,\n\n    parallel_safe: Option<syn::LitBool>,\n\n    transition_fn: AggregateFn,\n    final_fn: AggregateFn,\n\n    serialize_fn: Option<AggregateFn>,\n    deserialize_fn: Option<AggregateFn>,\n    combine_fn: Option<AggregateFn>,\n}\n\nenum AggregateItem {\n    State(AggregateTy),\n    Fn(AggregateFn),\n    ParallelSafe(AggregateParallelSafe),\n}\n\nstruct AggregateTy {\n    ident: syn::Ident,\n    ty: Box<syn::Type>,\n}\n\nstruct AggregateParallelSafe {\n    value: syn::LitBool,\n}\n\nstruct AggregateFn {\n    ident: syn::Ident,\n    sql_name: Option<syn::LitStr>,\n    parens: syn::token::Paren,\n    args: Punctuated<AggregateArg, Comma>,\n    ret: syn::ReturnType,\n    body: syn::Block,\n    fcinfo: Option<AggregateArg>,\n}\n\n#[derive(Clone)]\nstruct AggregateArg {\n    rust: syn::PatType,\n    sql: Option<syn::LitStr>,\n}\n\nmacro_rules! error {\n    ($span: expr, $fmt: literal, $($arg:expr),* $(,)?) => {\n        return Err(syn::Error::new($span, format!($fmt, $($arg),*)))\n    };\n    ($span: expr, $msg: literal) => {\n        return Err(syn::Error::new($span, $msg))\n    };\n}\n\nmacro_rules! check_duplicate {\n    ($val: expr, $span:expr, $name: expr) => {\n        if $val.is_some() {\n            error!($span, \"duplicate {}\")\n        }\n    };\n}\n\nimpl Parse for Aggregate {\n    fn parse(input: ParseStream) -> syn::Result<Self> {\n        let _: Token![impl] = input.parse()?;\n\n        let first_path_segment = input.parse()?;\n        let (schema, name): (_, syn::Ident) = if input.peek(Token![::]) {\n            let _: Token![::] = input.parse()?;\n            (Some(first_path_segment), input.parse()?)\n        } else {\n            (None, first_path_segment)\n        };\n\n        let body;\n        let _brace_token = syn::braced!(body in input);\n        let mut state_ty = None;\n\n        let mut parallel_safe = None;\n\n        let mut fns: Vec<AggregateFn> = vec![];\n        while !body.is_empty() {\n            use AggregateItem::*;\n            let item = body.parse()?;\n            match item {\n                State(ty) => {\n                    if ty.ident != \"State\" {\n                        error!(\n                            ty.ident.span(),\n                            \"unexpected `type {}`, expected `State`\", ty.ident\n                        )\n                    }\n                    if state_ty.is_some() {\n                        error!(ty.ident.span(), \"duplicate `type State`\")\n                    }\n                    state_ty = Some(ty);\n                }\n                ParallelSafe(safe) => parallel_safe = Some(safe.value),\n                Fn(f) => {\n                    fns.push(f);\n                }\n            }\n        }\n\n        let mut transition_fn = None;\n        let mut final_fn = None;\n        let mut serialize_fn = None;\n        let mut deserialize_fn = None;\n        let mut combine_fn = None;\n        for f in fns {\n            if f.ident == \"transition\" {\n                check_duplicate!(transition_fn, f.ident.span(), \"`fn transition`\");\n                if f.args.is_empty() {\n                    error!(\n                        f.parens.span,\n                        \"transition function must have at least one argument\"\n                    )\n                }\n                for arg in f.args.iter().skip(1) {\n                    if arg.sql.is_none() {\n                        error!(arg.rust.span(), \"missing SQL type\")\n                    }\n                }\n                transition_fn = Some(f);\n            } else if f.ident == \"finally\" {\n                check_duplicate!(final_fn, f.ident.span(), \"`fn finally`\");\n                if f.args.len() != 1 {\n                    error!(\n                        f.parens.span,\n                        \"final function must have at one argument of type `Option<Inner<State>>`\"\n                    )\n                }\n                if f.args[0].sql.is_some() {\n                    error!(\n                        f.args[0].sql.span(),\n                        \"should not have SQL type, will be inferred\"\n                    )\n                }\n                final_fn = Some(f);\n            } else if f.ident == \"serialize\" {\n                check_duplicate!(serialize_fn, f.ident.span(), \"`fn serialize`\");\n                if f.args.len() != 1 {\n                    error!(\n                        f.parens.span,\n                        \"serialize function must have at one argument of type `Inner<State>`\"\n                    )\n                }\n                if f.args[0].sql.is_some() {\n                    error!(\n                        f.args[0].sql.span(),\n                        \"should not have SQL type, will be inferred\"\n                    )\n                }\n                serialize_fn = Some(f);\n            } else if f.ident == \"deserialize\" {\n                check_duplicate!(deserialize_fn, f.ident.span(), \"`fn deserialize`\");\n                if f.args.len() != 1 {\n                    error!(\n                        f.parens.span,\n                        \"deserialize function must have at one argument of type `bytea`\"\n                    )\n                }\n                if f.args[0].sql.is_some() {\n                    error!(\n                        f.args[0].sql.span(),\n                        \"should not have SQL type, will be inferred\"\n                    )\n                }\n                deserialize_fn = Some(f);\n            } else if f.ident == \"combine\" {\n                check_duplicate!(combine_fn, f.ident.span(), \"`fn combine`\");\n                if f.args.len() != 2 {\n                    error!(f.parens.span, \"deserialize function must have at one argument of type `Option<Inner<State>>`\")\n                }\n                for arg in &f.args {\n                    if arg.sql.is_some() {\n                        error!(arg.sql.span(), \"should not have SQL type, will be inferred\")\n                    }\n                }\n                combine_fn = Some(f)\n            } else {\n                error!(\n                    f.ident.span(),\n                    \"unexpected `fn {}`, expected one of `transition`, `finally`, `serialize`, `deserialize`, or `combine`\",\n                    f.ident\n                )\n            }\n        }\n\n        let state_ty = match state_ty {\n            Some(state_ty) => state_ty,\n            None => error!(name.span(), \"missing `type State = ...;`\"),\n        };\n\n        let transition_fn = match transition_fn {\n            Some(transition_fn) => transition_fn,\n            None => error!(name.span(), \"missing `fn transition`\"),\n        };\n\n        let final_fn = match final_fn {\n            Some(final_fn) => final_fn,\n            None => error!(name.span(), \"missing `fn final`\"),\n        };\n\n        Ok(Aggregate {\n            schema,\n            name,\n            state_ty,\n            parallel_safe,\n            transition_fn,\n            final_fn,\n            serialize_fn,\n            deserialize_fn,\n            combine_fn,\n        })\n    }\n}\n\nimpl Parse for AggregateItem {\n    fn parse(input: ParseStream) -> syn::Result<Self> {\n        let ahead = input.fork();\n        let _ = ahead.call(syn::Attribute::parse_outer)?;\n        let lookahead = ahead.lookahead1();\n        if lookahead.peek(Token![fn]) {\n            input.parse().map(AggregateItem::Fn)\n        } else if lookahead.peek(Token![type]) {\n            input.parse().map(AggregateItem::State)\n        } else if lookahead.peek(Token![const]) {\n            input.parse().map(AggregateItem::ParallelSafe)\n        } else {\n            Err(lookahead.error())\n        }\n    }\n}\n\nimpl Parse for AggregateTy {\n    fn parse(input: ParseStream) -> syn::Result<Self> {\n        let _: Token![type] = input.parse()?;\n        let ident = input.parse()?;\n        let _: Token![=] = input.parse()?;\n        let ty = Box::new(input.parse()?);\n        let _: Token![;] = input.parse()?;\n        Ok(Self { ident, ty })\n    }\n}\n\nimpl Parse for AggregateParallelSafe {\n    fn parse(input: ParseStream) -> syn::Result<Self> {\n        let _: Token![const] = input.parse()?;\n        let name: syn::Ident = input.parse()?;\n        if name != \"PARALLEL_SAFE\" {\n            error!(\n                name.span(),\n                \"unexpected const `{}` expected `PARALLEL_SAFE`\", name\n            )\n        }\n        let _: Token![:] = input.parse()?;\n        let ty: syn::Ident = input.parse()?;\n        if ty != \"bool\" {\n            error!(ty.span(), \"unexpected type `{}` expected `bool`\", ty)\n        }\n        let _: Token![=] = input.parse()?;\n        let value = input.parse()?;\n        let _: Token![;] = input.parse()?;\n        Ok(Self { value })\n    }\n}\n\nfn is_fcinfo(arg: &AggregateArg) -> bool {\n    if let syn::Type::Path(p) = &*arg.rust.ty {\n        for id in p.path.segments.iter() {\n            if id.ident == \"FunctionCallInfo\" {\n                return true;\n            }\n        }\n    }\n    false\n}\n\nimpl Parse for AggregateFn {\n    fn parse(input: ParseStream) -> syn::Result<Self> {\n        let mut attributes = input.call(syn::Attribute::parse_outer)?;\n        let _: Token![fn] = input.parse()?;\n        let ident = input.parse()?;\n\n        let contents;\n        let parens = syn::parenthesized!(contents in input);\n\n        let mut args = Punctuated::new();\n        let mut fcinfo = None;\n        while !contents.is_empty() {\n            let arg: AggregateArg = contents.parse()?;\n            if is_fcinfo(&arg) {\n                fcinfo = Some(arg);\n                if contents.is_empty() {\n                    break;\n                }\n                let _comma: Token![,] = contents.parse()?;\n                continue;\n            }\n            args.push(arg);\n            if contents.is_empty() {\n                break;\n            }\n            let comma: Token![,] = contents.parse()?;\n            args.push_punct(comma);\n        }\n\n        let ret = input.parse()?;\n        let body = input.parse()?;\n\n        let expected_path = parse_quote!(sql_name);\n        let sql_name = match take_attr(&mut attributes, &expected_path) {\n            None => None,\n            Some(attribute) => attribute.parse_args()?,\n        };\n        if !attributes.is_empty() {\n            error!(attributes[0].span(), \"unexpected attribute\")\n        }\n        Ok(Self {\n            ident,\n            sql_name,\n            parens,\n            args,\n            ret,\n            body,\n            fcinfo,\n        })\n    }\n}\n\nimpl Parse for AggregateArg {\n    fn parse(input: ParseStream) -> syn::Result<Self> {\n        let arg: syn::FnArg = input.parse()?;\n        let mut rust = match arg {\n            syn::FnArg::Typed(pat) => pat,\n            _ => error!(arg.span(), \"`self` is not a valid aggregate argument\"),\n        };\n        let sql = {\n            let expected_path = parse_quote!(sql_type);\n            let attribute = take_attr(&mut rust.attrs, &expected_path);\n            match attribute {\n                None => None,\n                Some(attribute) => attribute.parse_args()?,\n            }\n        };\n        Ok(Self { rust, sql })\n    }\n}\n\nfn take_attr(attrs: &mut Vec<syn::Attribute>, path: &syn::Path) -> Option<syn::Attribute> {\n    let idx = attrs.iter().enumerate().find(|(_, a)| &a.path == path);\n    match idx {\n        None => None,\n        Some((idx, _)) => {\n            let attribute = attrs.remove(idx);\n            Some(attribute)\n        }\n    }\n}\n\n//\n// Expander\n//\n\nfn expand(agg: Aggregate) -> TokenStream2 {\n    use std::fmt::Write;\n    let Aggregate {\n        schema,\n        name,\n        state_ty,\n        parallel_safe,\n        transition_fn,\n        final_fn,\n        serialize_fn,\n        deserialize_fn,\n        combine_fn,\n    } = agg;\n\n    let state_ty = state_ty.ty;\n\n    let transition_fns = transition_fn.transition_fn_tokens(&schema, &name);\n    let final_fns = final_fn.final_fn_tokens(&schema, &name);\n\n    let mut extension_sql_reqs = vec![\n        transition_fn.outer_ident(&name),\n        final_fn.outer_ident(&name),\n    ];\n\n    let schema_qualifier = match &schema {\n        Some(schema) => format!(\"{schema}.\"),\n        None => String::new(),\n    };\n    let mut create = format!(\"\\nCREATE AGGREGATE {schema_qualifier}{name} (\");\n    for (i, (name, arg)) in transition_fn.sql_args().enumerate() {\n        if i != 0 {\n            let _ = write!(&mut create, \", \");\n        }\n        if let Some(name) = name {\n            let _ = write!(&mut create, \"{name} \");\n        }\n        let _ = write!(&mut create, \"{arg}\");\n    }\n    let transition_fn_ident = transition_fn.outer_ident(&name);\n    let final_fn_ident = final_fn.outer_ident(&name);\n    let _ = write!(\n        &mut create,\n        \") (\\n    \\\n            stype = internal,\\n    \\\n            sfunc = {schema_qualifier}{transition_fn_ident},\\n    \\\n            finalfunc = {schema_qualifier}{final_fn_ident}\"\n    );\n\n    let parallel_safe = parallel_safe.map(|p| {\n        let value = p.value();\n        let _ = write!(\n            &mut create,\n            \",\\n    parallel = {}\",\n            if value { \"safe\" } else { \"unsafe\" }\n        );\n        let serialize_fn_check = value\n            .then(|| {\n                serialize_fn.as_ref().is_none().then(|| {\n                    quote_spanned!(p.span()=>\n                        compile_error!(\"parallel safety requires a `fn serialize()` also\");\n                    )\n                })\n            })\n            .flatten();\n        let deserialize_fn_check = value\n            .then(|| {\n                deserialize_fn.as_ref().is_none().then(|| {\n                    quote_spanned!(p.span()=>\n                        compile_error!(\"parallel safety requires a `fn deserialize()` also\");\n                    )\n                })\n            })\n            .flatten();\n        let combine_fn_check = value\n            .then(|| {\n                combine_fn.as_ref().is_none().then(|| {\n                    quote_spanned!(p.span()=>\n                        compile_error!(\"parallel safety requires a `fn combine()` also\");\n                    )\n                })\n            })\n            .flatten();\n        quote_spanned!(p.span()=>\n            #serialize_fn_check\n            #deserialize_fn_check\n            #combine_fn_check\n\n            #[allow(dead_code)]\n            pub const PARALLEL_SAFE: bool = #value;\n        )\n    });\n\n    let mut add_function =\n        |f: AggregateFn,\n         field: &str,\n         make_tokens: fn(&AggregateFn, &Option<syn::Ident>, &syn::Ident) -> TokenStream2| {\n            extension_sql_reqs.push(f.outer_ident(&name));\n            let _ = write!(\n                &mut create,\n                \",\\n    {} = {}{}\",\n                field,\n                schema_qualifier,\n                f.outer_ident(&name)\n            );\n            make_tokens(&f, &schema, &name)\n        };\n\n    let serialize_fns_check = serialize_fn.as_ref().xor(deserialize_fn.as_ref()).map(|_| {\n        let s = serialize_fn.as_ref().map(|f| {\n            quote_spanned!(f.ident.span()=>\n                compile_error!(\"`fn deserialize()` is also required\");\n            )\n        });\n        let d = deserialize_fn.as_ref().map(|f| {\n            quote_spanned!(f.ident.span()=>\n                compile_error!(\"`fn serialize()` is also required\");\n            )\n        });\n        quote!(#s #d)\n    });\n\n    let combine_fns_check1 = serialize_fn.as_ref().xor(combine_fn.as_ref()).map(|_| {\n        let s = serialize_fn.as_ref().map(|f| {\n            quote_spanned!(f.ident.span()=>\n                compile_error!(\"`fn combine()` is also required\");\n            )\n        });\n        let c = combine_fn.as_ref().map(|f| {\n            quote_spanned!(f.ident.span()=>\n                compile_error!(\"`fn serialize()` is also required\");\n            )\n        });\n        quote!(#s #c)\n    });\n\n    let combine_fns_check2 = combine_fn.as_ref().xor(deserialize_fn.as_ref()).map(|_| {\n        let s = combine_fn.as_ref().map(|f| {\n            quote_spanned!(f.ident.span()=>\n                compile_error!(\"`fn deserialize()` is also required\");\n            )\n        });\n        let d = deserialize_fn.as_ref().map(|f| {\n            quote_spanned!(f.ident.span()=>\n                compile_error!(\"`fn combine()` is also required\");\n            )\n        });\n        quote!(#s #d)\n    });\n\n    let serialize_fns =\n        serialize_fn.map(|f| add_function(f, \"serialfunc\", AggregateFn::serialize_fn_tokens));\n    let deserialize_fns =\n        deserialize_fn.map(|f| add_function(f, \"deserialfunc\", AggregateFn::deserialize_fn_tokens));\n    let combine_fns =\n        combine_fn.map(|f| add_function(f, \"combinefunc\", AggregateFn::combine_fn_tokens));\n\n    let _ = write!(&mut create, \"\\n);\\n\");\n\n    let extension_sql_name = format!(\"{name}_extension_sql\");\n\n    quote! {\n        pub mod #name {\n            use super::*;\n\n            pub type State = #state_ty;\n\n            #serialize_fns_check\n\n            #combine_fns_check1\n\n            #combine_fns_check2\n\n            #parallel_safe\n\n            #transition_fns\n\n            #final_fns\n            #serialize_fns\n            #deserialize_fns\n            #combine_fns\n\n            pgrx::extension_sql!(\n                #create,\n                name=#extension_sql_name,\n                requires=[#(#extension_sql_reqs),*],\n            );\n        }\n    }\n}\n\nimpl AggregateFn {\n    fn transition_fn_tokens(\n        &self,\n        schema: &Option<syn::Ident>,\n        aggregate_name: &syn::Ident,\n    ) -> TokenStream2 {\n        let outer_ident = self.outer_ident(aggregate_name);\n        let Self {\n            ident,\n            args,\n            body,\n            ret,\n            fcinfo,\n            ..\n        } = self;\n\n        let schema = schema.as_ref().map(|s| {\n            let s = format!(\"{s}\");\n            quote!(, schema = #s)\n        });\n\n        let input_ty = &*args[0].rust.ty;\n\n        let state_type_check = state_type_check_tokens(input_ty, Some(()));\n\n        let fcinfo_arg = if let Some(fcinfo) = fcinfo {\n            fcinfo.clone()\n        } else {\n            syn::parse_str::<AggregateArg>(\"__fcinfo: pg_sys::FunctionCallInfo\").unwrap()\n        };\n\n        let mut expanded_args = args.clone();\n        if let Some(fcinfo) = fcinfo {\n            let trailing = expanded_args.trailing_punct();\n            if !trailing {\n                expanded_args.push_punct(Comma::default());\n            }\n            expanded_args.push_value(fcinfo.clone());\n            if trailing {\n                expanded_args.push_punct(Comma::default());\n            }\n        }\n\n        let fcinfo_ident = arg_ident(&fcinfo_arg);\n\n        let arg_signatures = args\n            .iter()\n            .chain(std::iter::once(&fcinfo_arg))\n            .skip(1)\n            .map(|arg| &arg.rust);\n\n        let arg_vals: Punctuated<syn::Pat, Comma> =\n            expanded_args.iter().skip(1).map(arg_ident).collect();\n\n        let inner_arg_signatures = expanded_args.iter().map(|arg| &arg.rust);\n\n        let return_type_check = state_type_check_tokens(&ret_type(ret), Some(()));\n\n        // use different variables for these to ensure the type-check is called\n        let input_var = syn::Ident::new(\"__inner\", input_ty.span());\n        let input_state_var = syn::Ident::new(\"state\", input_ty.span());\n\n        let input_type_check = quote_spanned!(input_ty.span()=>\n            let inner: Option<State> = match &mut #input_var {\n                None => None,\n                Some(inner) => Option::take(&mut **inner),\n            };\n            let #input_state_var: #input_ty = inner;\n        );\n\n        // use different variables for these to ensure the type-check is called\n        let result_var = syn::Ident::new(\"result\", ret_type(ret).span());\n        let state_var = syn::Ident::new(\"state\", ret_type(ret).span());\n        let result_type_check = quote_spanned!(state_var.span()=>\n            let #state_var: Option<State> = #result_var;\n        );\n\n        quote! {\n            #state_type_check\n            #return_type_check\n\n            #[pgrx::pg_extern(immutable, parallel_safe #schema)]\n            pub fn #outer_ident(\n                #input_var: pgrx::Internal,\n                #(#arg_signatures,)*\n            ) -> Option<Internal> {\n                use crate::palloc::{Inner, InternalAsValue, ToInternal};\n                unsafe {\n                    let mut #input_var: Option<Inner<Option<State>>> = #input_var.to_inner();\n                    #input_type_check\n                    crate::aggregate_utils::in_aggregate_context(#fcinfo_ident, || {\n                        let #result_var = #ident(#input_state_var, #arg_vals);\n                        #result_type_check\n\n                        #input_var = match (#input_var, state) {\n                            (None, None) => None,\n                            (None, state @ Some(..)) => {\n                                Some(state.into())\n                            },\n                            (Some(mut inner), state) => {\n                                *inner = state;\n                                Some(inner)\n                            },\n                        };\n                        #input_var.internal()\n                    })\n                }\n            }\n\n            pub fn #ident(#(#inner_arg_signatures),*) #ret\n                #body\n        }\n    }\n\n    fn final_fn_tokens(\n        &self,\n        schema: &Option<syn::Ident>,\n        aggregate_name: &syn::Ident,\n    ) -> TokenStream2 {\n        let outer_ident = self.outer_ident(aggregate_name);\n        let Self {\n            ident,\n            args,\n            ret,\n            body,\n            ..\n        } = self;\n\n        let schema = schema.as_ref().map(|s| {\n            let s = format!(\"{s}\");\n            quote!(, schema = #s)\n        });\n\n        let input_ty = &*args[0].rust.ty;\n\n        let state_type_check = type_check_tokens(input_ty, parse_quote!(Option<&mut State>));\n\n        let arg_vals: Punctuated<syn::Pat, Comma> = args.iter().skip(1).map(arg_ident).collect();\n\n        let inner_arg_signatures = args.iter().map(|arg| &arg.rust);\n\n        // use different variables for these to ensure the type-check is called\n        let input_var = syn::Ident::new(\"input\", input_ty.span());\n        let state_var = syn::Ident::new(\"state\", input_ty.span());\n        let input_type_check = quote_spanned!(input_ty.span()=>\n            let #state_var: #input_ty = #input_var;\n        );\n\n        quote! {\n            #state_type_check\n\n            #[pgrx::pg_extern(immutable, parallel_safe #schema)]\n            pub fn #outer_ident(\n                __internal: pgrx::Internal,\n                __fcinfo: pg_sys::FunctionCallInfo\n            ) #ret {\n                use crate::palloc::InternalAsValue;\n                unsafe {\n                    let mut #input_var: Option<Inner<Option<State>>> = __internal.to_inner();\n                    let #input_var: Option<&mut State> = #input_var.as_deref_mut()\n                        .map(|i| i.as_mut()).flatten();\n                    #input_type_check\n                    #ident(#state_var, #arg_vals)\n                }\n            }\n\n            pub fn #ident(#(#inner_arg_signatures,)*) #ret\n                #body\n        }\n    }\n\n    fn serialize_fn_tokens(\n        &self,\n        schema: &Option<syn::Ident>,\n        aggregate_name: &syn::Ident,\n    ) -> TokenStream2 {\n        let outer_ident = self.outer_ident(aggregate_name);\n        let Self {\n            ident,\n            args,\n            ret,\n            body,\n            ..\n        } = self;\n\n        let schema = schema.as_ref().map(|s| {\n            let s = format!(\"{s}\");\n            quote!(, schema = #s)\n        });\n\n        let input_ty = &*args[0].rust.ty;\n        let state_type_check = refstate_type_check_tokens(input_ty, None);\n\n        let return_type_check = bytea_type_check_tokens(&ret_type(ret));\n\n        let inner_arg_signatures = args.iter().map(|arg| &arg.rust);\n\n        // use different variables for these to ensure the type-check is called\n        let input_var = syn::Ident::new(\"input\", input_ty.span());\n        let state_var = syn::Ident::new(\"state\", input_ty.span());\n        let input_type_check = quote_spanned!(input_ty.span()=>\n            let #state_var: #input_ty = #input_var;\n        );\n\n        quote! {\n            #state_type_check\n\n            #return_type_check\n\n            #[pgrx::pg_extern(strict, immutable, parallel_safe #schema)]\n            pub fn #outer_ident(\n                __internal: pgrx::Internal,\n            ) -> bytea {\n                use crate::palloc::{Inner, InternalAsValue};\n                let #input_var: Option<Inner<Option<State>>> = unsafe {\n                    __internal.to_inner()\n                };\n                let mut #input_var: Inner<Option<State>> = #input_var.unwrap();\n                let #input_var: &mut State = #input_var.as_mut().unwrap();\n                #input_type_check\n                #ident(#state_var)\n            }\n\n            #[allow(clippy::ptr_arg)]\n            pub fn #ident(#(#inner_arg_signatures,)*)\n            -> bytea\n                #body\n        }\n    }\n\n    fn deserialize_fn_tokens(\n        &self,\n        schema: &Option<syn::Ident>,\n        aggregate_name: &syn::Ident,\n    ) -> TokenStream2 {\n        let outer_ident = self.outer_ident(aggregate_name);\n        let Self {\n            ident,\n            args,\n            ret,\n            body,\n            ..\n        } = self;\n\n        let schema = schema.as_ref().map(|s| {\n            let s = format!(\"{s}\");\n            quote!(, schema = #s)\n        });\n\n        let state_name = arg_ident(&args[0]);\n\n        let state_type_check = bytea_type_check_tokens(&args[0].rust.ty);\n\n        let return_type_check = state_type_check_tokens(&ret_type(ret), None);\n\n        // use different variables for these to ensure the type-check is called\n        let result_var = syn::Ident::new(\"result\", ret_type(ret).span());\n        let state_var = syn::Ident::new(\"state\", ret_type(ret).span());\n        let result_type_check = quote_spanned!(state_var.span()=>\n            let #state_var: State = #result_var;\n        );\n\n        // int8_avg_deserialize allocates in CurrentMemoryContext, so we do the same\n        // https://github.com/postgres/postgres/blob/f920f7e799c587228227ec94356c760e3f3d5f2b/src/backend/utils/adt/numeric.c#L5728-L5770\n        quote! {\n            #state_type_check\n\n            #return_type_check\n\n            #[pgrx::pg_extern(strict, immutable, parallel_safe #schema)]\n            pub fn #outer_ident(\n                bytes: crate::raw::bytea,\n                _internal: Internal\n            ) -> Option<Internal> {\n                use crate::palloc::ToInternal;\n                let #result_var = #ident(bytes);\n                #result_type_check\n                let state: Inner<Option<State>> = Some(state).into();\n                unsafe {\n                    Some(state).internal()\n                }\n            }\n\n            pub fn #ident(#state_name: crate::raw::bytea) #ret\n                #body\n        }\n    }\n\n    fn combine_fn_tokens(\n        &self,\n        schema: &Option<syn::Ident>,\n        aggregate_name: &syn::Ident,\n    ) -> TokenStream2 {\n        let outer_ident = self.outer_ident(aggregate_name);\n        let Self {\n            ident,\n            args,\n            ret,\n            body,\n            ..\n        } = self;\n\n        let schema = schema.as_ref().map(|s| {\n            let s = format!(\"{s}\");\n            quote!(, schema = #s)\n        });\n\n        let a_name = arg_ident(&args[0]);\n        let b_name = arg_ident(&args[1]);\n\n        let state_type_check_a = refstate_type_check_tokens(&args[0].rust.ty, Some(()));\n        let state_type_check_b = refstate_type_check_tokens(&args[1].rust.ty, Some(()));\n\n        let return_type_check = state_type_check_tokens(&ret_type(ret), Some(()));\n        let inner_arg_signatures = args.iter().map(|arg| &arg.rust);\n\n        // use different variables for these to ensure the type-check is called\n        let result_var = syn::Ident::new(\"result\", ret_type(ret).span());\n        let state_var = syn::Ident::new(\"state\", ret_type(ret).span());\n        let result_type_check = quote_spanned!(state_var.span()=>\n            let #state_var: Option<State> = #result_var;\n        );\n\n        let mod_counters = make_mod_counters();\n\n        quote! {\n            #state_type_check_a\n            #state_type_check_b\n            #return_type_check\n            #mod_counters\n\n            #[pgrx::pg_extern(immutable, parallel_safe #schema)]\n            pub fn #outer_ident(\n                #a_name: Internal,\n                #b_name: Internal,\n                __fcinfo: pg_sys::FunctionCallInfo\n            ) -> Option<Internal> {\n                use crate::palloc::{Inner, InternalAsValue, ToInternal};\n                unsafe {\n                    crate::aggregate_utils::in_aggregate_context(__fcinfo, || {\n                        let a: Option<Inner<State>> = #a_name.to_inner();\n                        let b: Option<Inner<State>> = #b_name.to_inner();\n                        #[cfg(any(test, feature = \"pg_test\"))]\n                        #aggregate_name::counters::increment_combine(&a, &b);\n                        let #result_var = #ident(\n                            a.as_deref(),\n                            b.as_deref(),\n                        );\n                        #result_type_check\n                        let state = match #state_var {\n                            None => None,\n                            state @ Some(..) => {\n                                let state: Inner<Option<State>> = state.into();\n                                Some(state)\n                            },\n                        };\n                        state.internal()\n                    })\n                }\n            }\n\n            #[allow(clippy::ptr_arg)]\n            pub fn #ident(#(#inner_arg_signatures,)*) #ret\n                #body\n        }\n    }\n\n    fn outer_ident(&self, aggregate_name: &syn::Ident) -> syn::Ident {\n        let name = match &self.sql_name {\n            Some(name) => name.value(),\n            None => format!(\"{}_{}_fn_outer\", aggregate_name, self.ident),\n        };\n        syn::Ident::new(&name, Span::call_site())\n    }\n\n    fn sql_args(&self) -> impl Iterator<Item = (Option<&syn::Ident>, String)> {\n        self.args.iter().skip(1).map(|arg| {\n            let ident = match &*arg.rust.pat {\n                syn::Pat::Ident(id) => Some(&id.ident),\n                _ => None,\n            };\n            (ident, arg.sql.as_ref().expect(\"missing sql arg\").value())\n        })\n    }\n}\n\nfn arg_ident(arg: &AggregateArg) -> syn::Pat {\n    syn::Pat::clone(&*arg.rust.pat)\n}\n\nfn make_mod_counters() -> TokenStream2 {\n    quote! {\n        #[cfg(any(test, feature = \"pg_test\"))]\n        pub mod counters {\n            use ::std::sync::atomic::{AtomicUsize, Ordering::Relaxed};\n            use crate::palloc::Inner;\n\n            pub static COMBINE_NONE: AtomicUsize = AtomicUsize::new(0);\n            pub static COMBINE_A: AtomicUsize = AtomicUsize::new(0);\n            pub static COMBINE_B: AtomicUsize = AtomicUsize::new(0);\n            pub static COMBINE_BOTH: AtomicUsize = AtomicUsize::new(0);\n\n            // Works as long as only one pg_test is run at a time.  If we have two\n            // running in the same process, need a mutex to ensure only one test is\n            // using the counters at a time.  Otherwise, a test may see non-zero\n            // counters because of another test's work rather than its own.\n            pub fn reset() {\n                COMBINE_NONE.store(0, Relaxed);\n                COMBINE_A.store(0, Relaxed);\n                COMBINE_B.store(0, Relaxed);\n                COMBINE_BOTH.store(0, Relaxed);\n            }\n\n            pub fn increment_combine<T>(a: &Option<Inner<T>>, b: &Option<Inner<T>>) {\n                match (a, b) {\n                    // TODO Remove COMBINE_NONE?  We suspect postgres never calls with (None, None); what would be the point?\n                    (None, None) => COMBINE_NONE.fetch_add(1, Relaxed),\n                    // TODO Remove COMBIINE_A?  We suspect postgres never calls with (Some, None), only (None, Some).\n                    (Some(_), None) => COMBINE_A.fetch_add(1, Relaxed),\n                    (None, Some(_)) => COMBINE_B.fetch_add(1, Relaxed),\n                    (Some(_), Some(_)) => COMBINE_BOTH.fetch_add(1, Relaxed),\n                };\n            }\n        }\n    }\n}\n\nfn ret_type(ret: &syn::ReturnType) -> Cow<'_, syn::Type> {\n    match ret {\n        syn::ReturnType::Default => Cow::Owned(parse_quote!(())),\n        syn::ReturnType::Type(_, ty) => Cow::Borrowed(ty),\n    }\n}\n\nfn state_type_check_tokens(ty: &syn::Type, optional: Option<()>) -> TokenStream2 {\n    match optional {\n        Some(..) => type_check_tokens(ty, parse_quote!(Option<State>)),\n        None => type_check_tokens(ty, parse_quote!(State)),\n    }\n}\n\nfn refstate_type_check_tokens(ty: &syn::Type, optional: Option<()>) -> TokenStream2 {\n    match optional {\n        Some(..) => type_check_tokens(ty, parse_quote!(Option<&State>)),\n        None => {\n            // we need to allow both &State and &mut State, so we use a\n            // different equality-checker for this case than the others\n            quote_spanned! {ty.span()=>\n                const _: () = {\n                    trait RefType {\n                        type Referenced;\n                    }\n                    impl<'a, T> RefType for &'a T { type Referenced = T; }\n                    impl<'a, T> RefType for &'a mut T { type Referenced = T; }\n                    fn check<T: RefType<Referenced=State>>() {}\n                    let _checked = check::<#ty>;\n                };\n            }\n        }\n    }\n}\n\nfn bytea_type_check_tokens(ty: &syn::Type) -> TokenStream2 {\n    type_check_tokens(ty, parse_quote!(bytea))\n}\n\nfn type_check_tokens(user_ty: &syn::Type, expected_type: syn::Type) -> TokenStream2 {\n    quote_spanned! {user_ty.span()=>\n        const _: () = {\n            trait SameType {\n                type This;\n            }\n            impl<T> SameType for T { type This = Self; }\n            fn check_type<T, U: SameType<This=T>>() {}\n            let _checked = check_type::<#expected_type, #user_ty>;\n        };\n    }\n}\n"
  },
  {
    "path": "crates/asap/Cargo.toml",
    "content": "[package]\nname = \"asap\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\n"
  },
  {
    "path": "crates/asap/src/fft.rs",
    "content": "// based on https://github.com/stanford-futuredata/ASAP/blob/8b39db4bc92590cbe5b44ddace9b7bb1d677248b/ASAP-optimized.js\n// original copyright notice as follows\n//\n// Free FFT and convolution (JavaScript)\n//\n// Copyright (c) 2014 Project Nayuki\n// https://www.nayuki.io/page/free-small-fft-in-multiple-languages\n//\n// (MIT License)\n// Permission is hereby granted, free of charge, to any person obtaining a copy of\n// this software and associated documentation files (the \"Software\"), to deal in\n// the Software without restriction, including without limitation the rights to\n// use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of\n// the Software, and to permit persons to whom the Software is furnished to do so,\n// subject to the following conditions:\n// - The above copyright notice and this permission notice shall be included in\n//   all copies or substantial portions of the Software.\n// - The Software is provided \"as is\", without warranty of any kind, express or\n//   implied, including but not limited to the warranties of merchantability,\n//   fitness for a particular purpose and noninfringement. In no event shall the\n//   authors or copyright holders be liable for any claim, damages or other\n//   liability, whether in an action of contract, tort or otherwise, arising from,\n//   out of or in connection with the Software or the use or other dealings in the\n//   Software.\n\n// TODO JOSH it looks like they have a rust version as well,\n//           we likely should be using that instead\n\nuse std::f64::consts::PI;\n\n/*\n * Computes the discrete Fourier transform (DFT) of the given complex vector, storing the result back into the vector.\n * The vector can have any length. This is a wrapper function.\n */\npub fn transform(real: &mut [f64], imag: &mut [f64]) {\n    assert_eq!(real.len(), imag.len());\n\n    let n = real.len();\n    if n == 0 {\n    } else if n & (n - 1) == 0 {\n        // Is power of 2\n        transform_radix2(real, imag);\n    } else {\n        // More complicated algorithm for arbitrary sizes\n        transform_bluestein(real, imag);\n    }\n}\n\n/*\n * Computes the inverse discrete Fourier transform (IDFT) of the given complex vector, storing the result back into the vector.\n * The vector can have any length. This is a wrapper function. This transform does not perform scaling, so the inverse is not a true inverse.\n */\npub fn inverse_transform(real: &mut [f64], imag: &mut [f64]) {\n    transform(imag, real);\n}\n\n/*\n * Computes the discrete Fourier transform (DFT) of the given complex vector, storing the result back into the vector.\n * The vector's length must be a power of 2. Uses the Cooley-Tukey decimation-in-time radix-2 algorithm.\n */\nfn transform_radix2(real: &mut [f64], imag: &mut [f64]) {\n    // Initialization\n    let n = real.len();\n    if n == 1 {\n        // Trivial transform\n        return;\n    }\n    let mut levels = 100;\n    for i in 0..32 {\n        if 1 << i == n {\n            levels = i; // Equal to log2(n)\n        }\n    }\n    debug_assert!(levels < 32);\n\n    let mut cos_table = vec![0.0; n / 2];\n    let mut sin_table = vec![0.0; n / 2];\n    for i in 0..n / 2 {\n        cos_table[i] = (2.0 * PI * i as f64 / n as f64).cos();\n        sin_table[i] = (2.0 * PI * i as f64 / n as f64).sin();\n    }\n\n    // Bit-reversed addressing permutation\n    for i in 0..n {\n        let j = reverse_bits(i as u32, levels) as usize;\n        if j > i {\n            real.swap(i, j);\n            imag.swap(i, j);\n        }\n    }\n\n    // Cooley-Tukey decimation-in-time radix-2 FFT\n    let mut size = 2;\n    while size <= n {\n        let halfsize = size / 2;\n        let tablestep = n / size;\n        for i in (0..n).step_by(size) {\n            let mut j = i;\n            let mut k = 0;\n            while j < i + halfsize {\n                let tpre = real[j + halfsize] * cos_table[k] + imag[j + halfsize] * sin_table[k];\n                let tpim = -real[j + halfsize] * sin_table[k] + imag[j + halfsize] * cos_table[k];\n                real[j + halfsize] = real[j] - tpre;\n                imag[j + halfsize] = imag[j] - tpim;\n                real[j] += tpre;\n                imag[j] += tpim;\n                j += 1;\n                k += tablestep;\n            }\n        }\n        size *= 2;\n    }\n\n    // Returns the integer whose value is the reverse of the lowest 'bits' bits of the integer 'x'.\n    fn reverse_bits(x: u32, bits: u32) -> u32 {\n        let mut x = x;\n        let mut y = 0;\n        for _ in 0..bits {\n            y = (y << 1) | (x & 1);\n            x >>= 1;\n        }\n        y\n    }\n}\n\n/*\n * Computes the discrete Fourier transform (DFT) of the given complex vector, storing the result back into the vector.\n * The vector can have any length. This requires the convolution function, which in turn requires the radix-2 FFT function.\n * Uses Bluestein's chirp z-transform algorithm.\n */\nfn transform_bluestein(real: &mut [f64], imag: &mut [f64]) {\n    // Find a power-of-2 convolution length m such that m >= n * 2 + 1\n    let n = real.len();\n    let mut m = 1;\n    while m < n * 2 + 1 {\n        m *= 2;\n    }\n\n    // Trignometric tables\n    let mut cos_table = vec![0.0; n];\n    let mut sin_table = vec![0.0; n];\n    for i in 0..n {\n        let j = (i * i % (n * 2)) as f64; // This is more accurate than j = i * i\n        cos_table[i] = (PI * j / n as f64).cos();\n        sin_table[i] = (PI * j / n as f64).sin();\n    }\n\n    // Temporary vectors and preprocessing\n    let mut areal = vec![0.0; m];\n    let mut aimag = vec![0.0; m];\n    for i in 0..n {\n        areal[i] = real[i] * cos_table[i] + imag[i] * sin_table[i];\n        aimag[i] = -real[i] * sin_table[i] + imag[i] * cos_table[i];\n    }\n    for i in n..m {\n        areal[i] = 0.0;\n        aimag[i] = 0.0;\n    }\n\n    let mut breal = vec![0.0; m];\n    let mut bimag = vec![0.0; m];\n    breal[0] = cos_table[0];\n    bimag[0] = sin_table[0];\n    for i in 1..n {\n        breal[i] = cos_table[i];\n        breal[m - i] = cos_table[i];\n        bimag[i] = sin_table[i];\n        bimag[m - i] = sin_table[i];\n    }\n    for i in n..=(m - n) {\n        breal[i] = 0.0;\n        bimag[i] = 0.0;\n    }\n\n    // Convolution\n    let mut creal = vec![0.0; m];\n    let mut cimag = vec![0.0; m];\n    convolve_complex(\n        &mut areal, &mut aimag, &mut breal, &mut bimag, &mut creal, &mut cimag,\n    );\n\n    // Postprocessing\n    for i in 0..n {\n        real[i] = creal[i] * cos_table[i] + cimag[i] * sin_table[i];\n        imag[i] = -creal[i] * sin_table[i] + cimag[i] * cos_table[i];\n    }\n}\n\n// /*\n//  * Computes the circular convolution of the given real vectors. Each vector's length must be the same.\n//  */\n// function convolveReal(x, y, out) {\n//     if (x.length != y.length || x.length != out.length)\n//         throw \"Mismatched lengths\";\n//     var zeros = new Array(x.length);\n//     for (var i = 0; i < zeros.length; i++)\n//         zeros[i] = 0;\n//     convolve_complex(x, zeros, y, zeros.slice(), out, zeros.slice());\n// }\n\n// /*\n//  * Computes the circular convolution of the given complex vectors. Each vector's length must be the same.\n//  */\nfn convolve_complex(\n    xreal: &mut [f64],\n    ximag: &mut [f64],\n    yreal: &mut [f64],\n    yimag: &mut [f64],\n    outreal: &mut [f64],\n    outimag: &mut [f64],\n) {\n    let n = xreal.len();\n\n    transform(xreal, ximag);\n    transform(yreal, yimag);\n    for i in 0..n {\n        let temp = xreal[i] * yreal[i] - ximag[i] * yimag[i];\n        ximag[i] = ximag[i] * yreal[i] + xreal[i] * yimag[i];\n        xreal[i] = temp;\n    }\n    inverse_transform(xreal, ximag);\n    for i in 0..n {\n        // Scaling (because this FFT implementation omits it)\n        outreal[i] = xreal[i] / n as f64;\n        outimag[i] = ximag[i] / n as f64;\n    }\n}\n"
  },
  {
    "path": "crates/asap/src/lib.rs",
    "content": "// based on https://github.com/stanford-futuredata/ASAP/blob/8b39db4bc92590cbe5b44ddace9b7bb1d677248b/ASAP-optimized.js\n// original copyright notice as follows\n//\n// Free FFT and convolution (JavaScript)\n//\n// Copyright (c) 2014 Project Nayuki\n// https://www.nayuki.io/page/free-small-fft-in-multiple-languages\n//\n// (MIT License)\n// Permission is hereby granted, free of charge, to any person obtaining a copy of\n// this software and associated documentation files (the \"Software\"), to deal in\n// the Software without restriction, including without limitation the rights to\n// use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of\n// the Software, and to permit persons to whom the Software is furnished to do so,\n// subject to the following conditions:\n// - The above copyright notice and this permission notice shall be included in\n//   all copies or substantial portions of the Software.\n// - The Software is provided \"as is\", without warranty of any kind, express or\n//   implied, including but not limited to the warranties of merchantability,\n//   fitness for a particular purpose and noninfringement. In no event shall the\n//   authors or copyright holders be liable for any claim, damages or other\n//   liability, whether in an action of contract, tort or otherwise, arising from,\n//   out of or in connection with the Software or the use or other dealings in the\n//   Software.\n\nmod fft;\n\n// Smooth out the data to promote human readability, resolution is an upper bound on the number of points returned\npub fn asap_smooth(data: &[f64], resolution: u32) -> Vec<f64> {\n    use std::borrow::Cow;\n\n    let data = if data.len() > 2 * resolution as usize {\n        let period = (data.len() as f64 / resolution as f64) as u32;\n        Cow::Owned(sma(data, period, period))\n    } else {\n        Cow::Borrowed(data)\n    };\n\n    let mut acf = Acf::new(&data, (data.len() as f64 / 10.0).round() as u32);\n    let peaks = acf.find_peaks();\n    let mut metrics = Metrics::new(&data);\n    let original_kurt = metrics.kurtosis();\n    let mut min_obj = metrics.roughness();\n    let mut window_size = 1_u32;\n    let mut lb = 1;\n    let mut largest_feasible = -1_i32;\n    let mut tail = data.len() as u32 / 10;\n\n    for i in (0..peaks.len()).rev() {\n        let w = peaks[i];\n        if w < lb || w == 1 {\n            break;\n        } else if (1.0 - acf.correlations[w as usize]).sqrt() * window_size as f64\n            > (1.0 - acf.correlations[window_size as usize]).sqrt() * w as f64\n        {\n            continue;\n        }\n        let smoothed = sma(&data, w, 1);\n        metrics = Metrics::new(&smoothed);\n        if metrics.kurtosis() >= original_kurt {\n            let roughness = metrics.roughness();\n            if roughness < min_obj {\n                min_obj = roughness;\n                window_size = w;\n            }\n            let test_lb =\n                w as f64 * (acf.max_acf - 1.0).sqrt() / (acf.correlations[w as usize] - 1.0);\n            if test_lb > lb as f64 {\n                lb = test_lb.round() as u32;\n            }\n            if largest_feasible < 0 {\n                largest_feasible = i as i32;\n            }\n        }\n    }\n\n    if largest_feasible > 0 {\n        if largest_feasible < (peaks.len() - 2) as i32 {\n            tail = peaks[(largest_feasible + 1) as usize];\n        }\n        if peaks[largest_feasible as usize] + 1 > lb {\n            lb = peaks[largest_feasible as usize] + 1;\n        }\n    }\n\n    window_size = binary_search(lb, tail, &data, min_obj, original_kurt, window_size);\n    sma(&data, window_size, 1)\n}\n\nfn binary_search(\n    head: u32,\n    tail: u32,\n    data: &[f64],\n    min_obj: f64,\n    original_kurt: f64,\n    window_size: u32,\n) -> u32 {\n    let mut head = head;\n    let mut tail = tail;\n    let mut min_obj = min_obj;\n    let mut window_size = window_size;\n    while head <= tail {\n        let w = (head + tail).div_ceil(2);\n        let smoothed = sma(data, w, 1);\n        let metrics = Metrics::new(&smoothed);\n        if metrics.kurtosis() >= original_kurt {\n            /* Search second half if feasible */\n            let roughness = metrics.roughness();\n            if roughness < min_obj {\n                window_size = w;\n                min_obj = roughness;\n            }\n            head = w + 1;\n        } else {\n            /* Search first half */\n            tail = w - 1;\n        }\n    }\n    window_size\n}\n\nfn sma(data: &[f64], range: u32, slide: u32) -> Vec<f64> {\n    let mut window_start = 0;\n    let mut sum = 0.0;\n    let mut count = 0;\n    let mut values = Vec::new();\n\n    for (i, val) in data.iter().enumerate() {\n        sum += val;\n        count += 1;\n        if i + 1 - window_start >= range as usize {\n            values.push(sum / count as f64);\n            let old_start = window_start;\n            while window_start < data.len() && window_start - old_start < slide as usize {\n                sum -= data[window_start];\n                count -= 1;\n                window_start += 1;\n            }\n        }\n    }\n\n    values\n}\n\nfn mean(values: &[f64]) -> f64 {\n    values.iter().sum::<f64>() / values.len() as f64\n}\n\nfn std(values: &[f64]) -> f64 {\n    let m = mean(values);\n\n    let std: f64 = values.iter().map(|&x| (x - m).powi(2)).sum();\n    (std / values.len() as f64).sqrt()\n}\n\nimpl<'a> Acf<'a> {\n    fn new(values: &'a [f64], max_lag: u32) -> Acf<'a> {\n        let mut acf = Acf {\n            mean: mean(values),\n            values,\n            correlations: Vec::with_capacity(max_lag as usize),\n            max_acf: 0.0,\n        };\n        acf.calculate();\n        acf\n    }\n\n    fn calculate(&mut self) {\n        /* Padding to the closest power of 2 */\n        let len = (2_u32).pow((self.values.len() as f64).log2() as u32 + 1);\n        let mut fftreal = vec![0.0; len as usize];\n        let mut fftimg = vec![0.0; len as usize];\n\n        for (i, real) in fftreal.iter_mut().enumerate().take(self.values.len()) {\n            *real = self.values[i] - self.mean;\n        }\n\n        /* F_R(f) = FFT(X) */\n        fft::transform(&mut fftreal, &mut fftimg);\n\n        /* S(f) = F_R(f)F_R*(f) */\n        for i in 0..fftreal.len() {\n            fftreal[i] = fftreal[i].powi(2) + fftimg[i].powi(2);\n            fftimg[i] = 0.0;\n        }\n\n        /*  R(t) = IFFT(S(f)) */\n        fft::inverse_transform(&mut fftreal, &mut fftimg);\n        for i in 1..self.correlations.len() {\n            self.correlations[i] = fftreal[i] / fftreal[0];\n        }\n    }\n\n    fn find_peaks(&mut self) -> Vec<u32> {\n        const CORR_THRESH: f64 = 0.2;\n\n        let mut peak_indicies = Vec::new();\n\n        if self.correlations.len() > 1 {\n            let mut positive = self.correlations[1] > self.correlations[0];\n            let mut max = 1;\n            for i in 2..self.correlations.len() {\n                if !positive && self.correlations[i] > self.correlations[i - 1] {\n                    max = i;\n                    positive = !positive;\n                } else if positive && self.correlations[i] > self.correlations[max] {\n                    max = i;\n                } else if positive\n                    && self.correlations[i] < self.correlations[i - 1]\n                    && max > 1\n                    && self.correlations[max] > CORR_THRESH\n                {\n                    peak_indicies.push(max as u32);\n                    if self.correlations[max] > self.max_acf {\n                        self.max_acf = self.correlations[max];\n                    }\n                    positive = !positive;\n                }\n            }\n        }\n\n        /* If there is no autocorrelation peak within the MAX_WINDOW boundary,\n        # try windows from the largest to the smallest */\n\n        if peak_indicies.len() <= 1 {\n            for i in 2..self.correlations.len() {\n                peak_indicies.push(i as u32);\n            }\n        }\n\n        peak_indicies\n    }\n}\n\nstruct Metrics<'a> {\n    len: u32,\n    values: &'a [f64],\n    m: f64,\n}\n\nimpl Metrics<'_> {\n    fn new(values: &[f64]) -> Metrics<'_> {\n        Metrics {\n            len: values.len() as u32,\n            values,\n            m: mean(values),\n        }\n    }\n\n    fn kurtosis(&self) -> f64 {\n        let mut u4 = 0.0;\n        let mut variance = 0.0;\n\n        for value in self.values {\n            u4 += (value - self.m).powi(4);\n            variance += (value - self.m).powi(2);\n        }\n\n        self.len as f64 * u4 / variance.powi(2)\n    }\n\n    fn roughness(&self) -> f64 {\n        std(&self.diffs())\n    }\n\n    fn diffs(&self) -> Vec<f64> {\n        let mut diff = vec![0.0; (self.len - 1) as usize];\n        for i in 1..self.len as usize {\n            diff[i - 1] = self.values[i] - self.values[i - 1];\n        }\n        diff\n    }\n}\n\nstruct Acf<'a> {\n    mean: f64,\n    values: &'a [f64],\n    correlations: Vec<f64>,\n    max_acf: f64,\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn simple_sma_test() {\n        let data = vec![0.0, 1.0, 2.0, 3.0, 4.0];\n\n        let test = sma(&data, 3, 1);\n        assert_eq!(test, vec![1.0, 2.0, 3.0]);\n    }\n\n    #[test]\n    fn sma_slide_test() {\n        let data = vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0];\n\n        let test = sma(&data, 3, 2);\n        assert_eq!(test, vec![1.0, 3.0, 5.0]);\n    }\n\n    #[test]\n    fn sma_slide_unaliged_test() {\n        let data = vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0];\n\n        let test = sma(&data, 3, 2);\n        assert_eq!(test, vec![1.0, 3.0, 5.0]);\n    }\n\n    #[test]\n    fn sma_downsample_test() {\n        let data = vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0];\n\n        let test = sma(&data, 2, 2);\n        assert_eq!(test, vec![0.5, 2.5, 4.5, 6.5]);\n    }\n\n    #[test]\n    fn test_roughness_and_kurtosis() {\n        let series_a = vec![-1.0, 1.0, -1.0, 1.0, -1.0, 1.0]; // bimodal\n        let x = (3.0 - ((9.0 + 8.0 * 1.82) as f64).sqrt()) / 4.0; // ~ -0.45, calculated for specific mean and std\n        let series_b = vec![-1.0, -0.8, x, -0.2, 0.5, 1.5 - x]; // uneven but monotonically increasing\n        let x = ((1.0 / 2.0) as f64).sqrt();\n        let series_c = vec![-2.0 * x, -1.0 * x, 0.0, x, 2.0 * x]; // linear\n\n        assert_eq!(mean(&series_a), 0.0);\n        assert_eq!(std(&series_a), 1.0);\n        assert_eq!(mean(&series_b), 0.0);\n        assert_eq!(std(&series_b), 1.0);\n        assert!(mean(&series_c).abs() < 0.000000000001); // float precision breaks == 0 here\n        assert_eq!(std(&series_c), 1.0);\n\n        let test = Metrics::new(&series_a);\n        assert_eq!(\n            test.roughness(),\n            ((3.0 * 1.6_f64.powi(2) + 2.0 * 2.4_f64.powi(2)) / 5.0).sqrt()\n        );\n        assert_eq!(test.kurtosis(), 1.0);\n        let test = Metrics::new(&series_b);\n        assert_eq!(test.roughness(), 0.4686099077599554); // manually verified\n        assert!((test.kurtosis() - 2.7304).abs() < 0.000000000001); // = 2.7304\n        let test = Metrics::new(&series_c);\n        assert_eq!(test.roughness(), 0.0);\n        assert!((test.kurtosis() - 1.7).abs() < 0.000000000001); // == 1.7\n    }\n\n    #[test]\n    fn test_smoothing() {\n        // Monthly English temperature data from 1723 through 1970 (~3k pts)\n        #[rustfmt::skip]\n\tlet data = vec![1.1,4.4,7.5,8.9,11.7,15.0,15.3,15.6,13.3,11.1,7.5,5.8,5.6,4.2,4.7,7.2,11.4,15.3,15.0,16.2,14.4,8.6,5.3,3.3,4.4,3.3,5.0,8.1,10.8,12.2,13.8,13.3,12.8,9.4,6.9,3.9,1.1,4.2,4.2,8.4,13.4,16.4,16.0,15.6,14.7,10.2,6.1,1.8,4.2,5.0,5.1,9.2,13.6,14.9,16.9,16.9,14.4,10.8,4.7,3.6,3.9,2.4,7.1,8.3,12.5,16.4,16.9,16.0,12.8,9.1,7.2,1.6,1.2,2.3,2.8,7.1,10.3,15.1,16.8,15.7,16.6,10.1,8.1,5.0,4.1,4.7,6.2,8.7,12.4,14.0,15.3,16.3,15.3,10.9,9.2,3.4,1.9,2.2,6.0,6.8,12.1,15.6,16.3,16.7,15.3,12.3,7.8,5.2,2.4,6.4,6.1,8.9,11.4,14.6,16.0,16.6,14.5,10.9,6.3,2.2,6.9,6.0,5.9,10.0,11.2,15.2,18.3,16.1,12.8,9.1,6.5,7.6,4.3,6.4,8.1,9.3,11.1,14.1,16.2,16.2,13.3,8.4,6.2,4.0,4.4,4.0,5.8,8.9,10.9,13.3,14.8,16.2,14.2,10.3,6.3,5.4,6.4,3.1,6.9,8.6,10.6,15.7,16.4,17.8,14.4,10.4,6.9,6.4,6.2,4.2,6.1,8.8,12.5,15.9,17.4,13.8,14.2,8.9,6.1,4.9,4.6,4.6,5.5,9.9,11.4,14.2,16.4,16.0,12.5,10.2,6.3,6.1,4.0,6.8,5.8,6.7,11.6,15.2,16.0,14.7,13.1,9.6,3.7,3.2,2.8,1.6,3.9,6.4,8.6,12.8,15.3,14.7,14.0,5.3,3.3,2.2,1.7,4.4,4.2,7.1,9.3,15.2,15.6,16.7,14.7,11.0,7.8,3.9,1.9,3.6,4.1,6.6,10.6,15.0,15.8,15.8,12.2,9.2,4.4,1.1,3.6,5.4,5.3,5.4,13.3,15.6,14.9,16.9,14.2,8.9,9.3,4.9,1.4,2.9,4.8,6.7,10.8,14.4,16.4,15.4,12.8,9.4,6.9,3.5,3.8,2.3,4.4,7.5,11.4,12.2,16.1,15.0,14.2,10.3,5.8,2.7,2.5,1.4,3.1,6.9,12.8,14.3,15.8,15.9,14.2,7.8,3.3,5.3,3.3,5.8,2.5,8.1,12.2,14.7,16.9,18.3,14.4,9.4,6.9,5.3,2.5,1.8,1.8,6.3,10.4,14.8,15.4,15.8,14.2,9.2,7.1,6.0,5.3,3.6,5.3,6.8,12.3,11.9,17.2,15.6,13.8,10.1,6.7,4.7,4.0,6.7,8.2,7.7,10.7,14.2,17.2,15.0,15.2,9.2,4.0,4.2,4.0,1.5,6.2,7.1,9.3,14.9,15.3,14.6,12.6,8.3,4.3,3.0,3.2,3.1,5.6,6.8,10.3,14.8,15.6,15.7,13.9,10.6,6.5,4.2,2.2,3.6,6.0,7.5,12.1,14.6,15.2,15.7,13.1,10.0,4.6,4.4,3.3,2.8,3.4,6.7,12.2,13.6,14.7,15.7,13.9,10.4,5.7,3.6,2.2,1.2,3.9,10.0,9.4,15.7,15.0,14.6,13.5,8.4,4.7,3.9,4.4,4.6,6.0,6.7,9.1,13.8,16.1,14.7,13.6,9.4,3.9,2.9,0.3,4.0,4.9,8.1,10.7,14.0,18.4,15.2,13.3,8.2,7.1,3.2,2.6,3.8,5.2,7.2,13.8,14.6,14.2,16.4,11.9,8.1,5.7,3.9,5.9,5.8,6.1,8.6,12.1,15.0,18.2,16.3,13.5,10.9,5.1,2.5,1.9,3.8,6.6,9.4,11.7,15.2,16.9,15.8,15.7,9.2,5.7,6.1,5.4,5.8,6.8,9.4,11.9,14.3,15.8,16.4,14.2,9.4,6.2,4.4,4.7,4.0,3.7,10.0,12.9,16.9,17.8,15.3,13.6,7.9,4.6,3.6,0.8,4.9,5.4,8.9,10.2,14.6,15.3,15.3,13.1,8.3,5.8,6.2,3.7,3.8,3.9,7.2,12.2,13.9,16.1,15.2,12.5,8.9,4.4,2.8,4.8,0.4,5.0,7.5,11.4,13.8,15.7,15.3,13.3,9.2,3.9,1.7,0.7,1.7,4.2,8.1,9.7,13.7,15.7,16.6,13.3,9.3,7.2,3.3,0.1,5.4,4.7,7.3,10.0,12.8,14.4,16.1,14.1,9.2,6.9,3.3,0.8,4.8,4.7,8.1,12.2,13.9,15.6,16.0,11.7,9.2,5.6,4.6,2.5,2.7,5.0,7.8,11.3,13.1,16.4,15.0,12.8,8.2,5.7,4.8,3.7,4.6,2.5,5.4,10.0,13.1,15.3,15.8,13.9,8.9,5.3,3.6,1.0,3.2,3.1,5.5,12.2,14.3,15.7,14.3,12.2,9.2,6.3,5.6,1.2,1.9,4.4,6.4,10.1,16.1,16.9,16.1,13.0,11.7,7.2,4.8,4.0,2.6,6.5,8.3,10.3,14.7,15.9,17.2,12.4,9.9,5.3,3.8,0.6,4.3,6.4,8.6,10.9,14.7,16.1,16.1,12.5,10.3,4.8,3.5,4.6,6.1,6.0,9.8,12.6,16.6,16.7,15.8,14.3,9.3,4.8,4.5,1.6,3.8,6.4,9.4,10.8,14.1,16.3,15.2,12.9,10.2,6.2,4.4,1.9,2.3,6.8,7.2,11.7,13.6,15.3,15.9,14.6,10.2,6.9,2.6,1.9,3.2,4.6,8.2,10.6,15.4,17.3,16.8,12.2,7.4,6.7,6.1,2.9,7.9,7.9,9.4,11.9,14.4,17.9,17.6,15.2,10.9,5.7,3.1,0.9,2.1,7.9,6.3,12.8,14.2,16.8,17.6,15.6,9.1,4.4,3.2,2.1,4.8,6.6,9.2,12.1,16.2,17.4,17.3,14.2,10.6,6.5,5.4,5.2,1.9,4.1,5.2,9.0,14.9,15.6,14.2,13.3,7.6,2.3,2.8,3.4,3.3,3.3,10.1,10.4,14.8,18.8,15.8,12.8,9.8,6.2,2.7,0.6,1.4,2.7,5.7,13.5,13.7,15.2,14.0,14.8,7.8,5.5,0.3,3.4,0.4,1.2,8.4,12.3,16.1,16.1,13.9,13.6,8.7,5.6,2.8,2.7,3.4,2.1,8.1,11.2,16.1,15.0,15.1,11.7,7.5,3.3,2.8,3.6,5.9,6.8,7.4,11.5,13.9,15.8,15.6,12.8,9.8,4.5,3.8,3.9,3.8,3.6,9.4,13.8,15.4,15.8,15.8,13.4,9.8,6.1,0.3,1.5,5.0,2.1,7.4,12.5,14.0,15.4,16.6,13.1,8.6,4.6,6.1,4.3,6.6,6.4,6.1,11.9,14.6,14.9,15.6,12.2,10.3,6.1,4.3,4.3,4.7,6.5,9.6,10.6,14.7,15.3,15.9,13.8,8.9,5.9,1.1,2.3,4.5,5.9,10.0,10.2,13.2,15.3,16.9,11.8,8.8,7.1,4.3,2.8,4.6,4.2,6.2,10.9,13.5,17.6,15.0,11.7,11.3,6.0,5.3,1.8,7.2,7.0,10.2,11.3,15.7,18.1,15.5,12.5,9.6,6.1,3.7,3.1,0.8,3.9,7.7,10.9,13.2,15.2,16.6,16.0,11.7,4.5,6.6,7.3,4.7,4.2,10.2,10.3,13.9,14.7,15.9,14.6,8.1,4.6,0.3,3.5,4.6,4.3,7.4,11.3,13.6,17.3,15.8,12.5,8.2,4.7,4.8,3.6,4.0,5.1,10.4,12.9,16.9,16.3,16.4,13.6,9.9,4.7,1.5,1.7,2.8,3.4,5.4,9.6,14.1,15.2,14.4,12.9,8.3,5.6,1.3,2.8,2.2,4.0,9.3,12.3,13.9,17.7,16.8,13.9,9.2,5.4,3.3,4.6,4.8,6.7,8.3,12.1,14.8,16.1,17.1,14.2,10.2,4.8,1.5,1.6,3.7,5.6,8.9,10.6,13.7,13.5,17.2,13.8,10.1,5.1,3.6,1.8,3.4,6.3,9.1,10.5,13.7,17.6,16.1,11.4,9.3,5.0,4.4,5.8,2.9,4.7,6.9,13.3,16.1,15.9,15.6,14.2,10.7,6.6,2.1,2.1,4.1,6.2,8.3,10.2,13.2,16.0,16.4,14.6,8.2,4.8,3.6,4.2,4.3,5.1,6.8,12.1,14.9,15.4,16.2,13.4,10.6,7.8,6.8,2.8,3.7,2.9,7.7,11.8,14.2,17.1,16.9,10.5,11.4,2.9,1.8,2.6,2.8,3.2,5.8,13.7,14.8,18.4,16.7,12.7,7.2,6.0,2.2,2.0,5.7,6.0,5.2,13.1,13.7,15.1,14.8,12.7,10.2,4.6,4.1,2.2,3.5,4.9,8.2,9.2,14.6,15.2,14.6,13.9,9.8,5.4,3.6,1.2,4.6,7.1,8.9,12.8,14.1,16.1,14.4,13.7,12.3,7.7,3.1,2.6,5.3,3.5,5.5,10.9,13.0,14.2,14.3,13.2,9.3,4.9,1.7,1.9,5.8,6.8,7.6,11.6,13.6,15.0,14.5,12.5,8.1,4.3,2.8,2.9,1.4,2.9,9.6,9.2,12.2,16.0,14.7,12.8,8.1,4.7,4.3,0.3,6.5,7.3,8.1,12.6,14.3,14.9,15.3,13.4,10.3,3.4,2.3,2.7,2.1,3.9,6.6,9.9,12.8,13.4,13.9,11.8,10.3,3.9,3.1,4.5,6.4,5.5,7.6,8.7,15.1,14.1,13.6,13.2,6.4,9.1,2.5,4.4,2.7,4.5,6.9,11.3,16.4,18.2,15.3,13.3,12.0,9.5,3.6,4.4,4.3,6.8,8.6,11.5,13.4,16.4,17.4,13.4,9.1,4.1,1.4,0.3,3.2,4.7,8.9,11.4,13.6,15.7,14.7,12.3,8.1,5.6,4.7,3.6,2.1,5.7,9.5,9.4,12.3,14.8,16.4,14.9,10.4,8.6,6.4,4.7,6.3,7.8,8.3,12.7,17.1,15.6,15.2,12.4,10.7,8.2,1.6,0.1,3.1,5.0,6.7,12.2,12.3,14.1,14.4,12.5,8.4,7.1,4.8,4.3,4.7,4.6,7.4,10.7,13.4,16.0,15.1,13.7,9.5,7.2,5.1,3.8,3.9,5.0,9.1,11.6,14.1,17.2,16.3,15.1,10.8,5.2,4.6,0.4,6.4,6.3,8.8,11.2,17.3,17.9,17.6,13.6,11.1,4.4,5.8,1.7,0.7,5.9,8.9,11.9,14.2,16.5,14.8,13.7,11.4,6.9,6.9,5.1,5.2,6.6,8.3,12.4,15.4,16.0,15.3,14.3,10.2,7.4,7.4,0.3,4.3,4.3,6.7,12.5,14.9,15.1,14.3,11.3,8.3,4.5,1.4,0.2,2.2,7.7,8.9,12.0,12.7,16.2,13.7,11.9,10.4,6.9,1.9,1.6,4.8,7.2,9.2,11.5,15.4,16.7,16.9,13.7,12.7,5.6,5.8,3.1,3.4,5.8,8.6,10.9,15.2,15.9,15.4,13.6,10.7,5.9,5.1,1.2,5.6,3.9,7.7,15.1,14.6,15.8,14.3,12.1,10.1,6.6,6.9,7.1,5.6,7.1,7.7,13.0,15.4,16.9,16.2,13.8,10.6,6.7,5.6,2.9,5.7,5.8,8.6,11.3,15.0,16.4,16.9,13.4,8.9,6.6,3.1,3.7,3.5,5.8,7.2,11.1,15.3,15.4,14.6,11.7,8.6,5.3,4.1,2.7,4.7,2.3,4.7,9.9,15.5,16.9,15.7,12.5,10.5,5.2,5.3,1.5,0.4,4.9,6.1,10.5,14.4,15.6,15.1,12.7,9.8,4.6,4.0,2.8,4.1,4.2,6.4,10.2,14.3,14.9,14.6,12.4,9.3,7.3,3.7,4.1,3.6,3.8,9.7,11.4,14.1,13.8,15.9,11.1,7.5,11.4,1.3,1.1,2.4,7.5,7.8,12.7,12.9,13.8,14.6,13.4,8.7,5.2,4.4,0.6,4.2,6.4,7.8,11.4,15.6,14.5,17.1,13.2,7.2,5.5,7.2,4.0,1.9,5.6,8.5,10.4,12.8,14.8,15.3,14.4,7.9,5.7,7.4,3.8,1.6,4.7,9.8,10.8,14.7,15.4,13.5,13.3,9.2,5.9,0.4,3.2,0.9,2.0,8.6,9.5,14.9,14.3,13.5,11.4,9.5,6.7,4.6,6.3,6.4,6.1,7.8,12.3,18.2,16.5,16.6,14.7,9.5,6.9,0.5,2.2,2.4,5.6,6.6,12.3,13.9,17.5,15.2,11.5,10.7,7.9,4.8,1.3,6.1,5.9,8.2,13.9,14.5,15.6,13.6,12.8,9.7,5.8,5.6,3.9,5.7,6.1,6.4,12.1,13.9,15.4,15.6,13.3,9.2,6.6,3.4,0.7,6.4,4.7,9.0,10.1,15.4,16.2,14.5,12.3,7.9,7.4,4.6,5.6,4.7,5.8,7.4,10.4,14.3,14.6,15.5,12.7,10.8,3.1,4.8,4.9,4.7,5.2,8.2,10.6,13.2,18.7,15.8,12.9,7.8,7.9,7.7,5.1,0.6,3.4,7.6,10.9,14.3,14.9,14.7,12.3,10.1,5.2,1.3,3.6,4.3,6.7,9.2,10.3,13.2,15.4,15.2,14.4,9.4,4.9,5.1,2.4,1.7,3.3,7.1,8.8,13.3,16.8,15.7,13.2,9.7,5.3,2.4,3.7,5.3,4.4,8.1,9.4,13.5,15.3,16.9,12.5,10.7,4.8,4.4,2.6,4.3,5.3,7.5,11.3,15.8,16.4,17.4,14.5,11.3,7.2,7.3,3.4,1.8,4.9,7.7,10.8,16.8,14.8,15.8,14.7,9.6,4.3,4.8,4.9,5.7,7.3,7.5,11.7,14.8,18.3,16.2,12.9,9.6,4.8,1.6,3.5,1.7,4.7,5.9,11.5,12.3,14.5,13.6,11.2,9.8,4.5,1.5,1.7,4.9,6.3,7.5,10.4,14.8,15.0,15.9,13.1,11.8,4.2,3.9,3.9,5.1,5.5,8.8,12.3,12.7,14.2,14.6,13.0,10.3,3.2,6.5,4.9,5.8,6.5,8.8,10.6,13.6,15.2,15.5,11.6,9.9,7.3,6.3,2.4,2.3,4.7,8.8,12.4,13.6,15.5,14.2,13.3,9.8,5.5,3.7,2.1,2.3,2.9,10.6,12.6,15.6,16.6,15.1,16.3,9.7,6.7,5.8,5.8,4.4,4.8,8.6,10.0,15.5,15.5,14.7,12.8,10.7,6.9,6.1,1.2,6.9,3.1,9.3,11.4,14.1,14.9,16.3,13.6,9.3,4.8,3.4,3.9,6.3,6.8,8.7,13.5,15.5,18.3,16.8,14.3,8.4,4.9,7.2,5.6,7.5,3.8,10.1,9.6,13.2,17.3,15.5,14.4,9.7,5.8,2.9,3.3,2.8,4.7,9.2,11.7,15.2,17.5,15.7,12.9,9.5,4.7,0.6,0.5,6.1,7.3,8.7,11.3,12.8,15.2,17.2,12.7,9.8,3.4,3.6,5.0,6.9,6.8,8.2,9.7,14.1,17.1,15.3,13.2,8.4,7.0,5.3,5.2,1.8,5.4,7.7,9.9,14.2,16.2,15.4,11.8,8.6,6.3,5.3,5.5,3.9,6.7,9.8,10.0,13.9,17.3,15.1,13.6,10.4,5.6,0.2,6.4,2.3,5.1,8.6,12.3,14.2,14.8,16.1,14.9,8.9,5.4,4.2,3.2,4.8,4.5,7.9,9.6,14.3,17.2,16.5,12.7,11.3,6.1,6.0,5.5,6.2,4.9,7.0,9.1,15.2,14.7,15.2,11.2,9.3,7.1,4.7,4.6,5.6,5.4,8.9,11.8,15.1,16.6,16.2,13.3,10.2,3.5,0.3,0.7,3.1,4.7,5.7,8.9,12.9,13.6,14.5,12.6,8.9,4.1,0.7,0.9,5.8,6.2,7.9,10.4,13.8,15.5,16.4,14.6,7.1,5.4,5.1,1.5,3.2,5.3,7.3,11.8,13.7,16.2,13.9,12.7,7.3,8.9,3.9,5.2,6.1,7.4,8.4,11.5,13.1,15.2,14.9,12.1,9.9,5.7,3.9,4.7,5.9,1.8,8.1,10.6,13.9,14.5,15.3,13.3,9.7,5.8,4.6,6.5,5.3,6.5,7.2,11.3,14.1,16.3,17.2,14.5,9.4,5.3,4.4,2.9,5.8,4.5,7.7,8.9,13.9,16.3,13.6,12.2,7.5,5.9,3.7,2.1,1.5,4.2,7.6,10.3,13.5,15.9,15.8,13.6,11.3,6.6,1.9,2.4,3.8,3.3,6.2,9.4,15.3,17.3,15.7,11.8,7.1,4.4,2.6,3.2,1.8,3.1,6.2,10.7,13.2,13.7,14.1,12.2,7.9,7.7,4.9,3.4,2.9,4.7,7.1,12.9,15.3,15.3,14.7,12.8,8.6,6.9,3.3,5.7,3.1,6.2,7.1,11.7,13.5,14.5,14.1,14.6,9.4,5.7,0.8,1.3,3.9,3.8,6.2,9.5,14.7,15.1,14.1,14.2,9.4,5.6,4.1,2.3,3.6,2.7,7.3,11.6,13.4,14.3,15.2,12.4,7.1,6.4,1.8,2.2,4.7,7.2,10.3,13.1,15.6,16.4,17.4,12.9,9.9,5.2,4.8,3.4,5.1,6.7,9.7,9.2,13.5,15.9,14.2,11.6,9.3,7.9,5.1,0.2,1.8,5.1,8.2,12.4,14.8,15.2,15.8,15.4,7.1,7.5,3.9,4.8,4.6,6.7,9.1,11.9,16.2,16.2,14.3,13.1,6.9,4.3,3.9,1.6,5.8,6.5,7.1,10.0,15.1,16.5,16.2,12.1,9.9,7.6,4.7,6.6,4.8,4.3,8.5,10.2,13.6,15.3,16.5,15.2,11.3,7.2,7.3,4.9,5.1,5.1,7.8,9.9,15.7,17.3,17.8,13.2,8.8,8.5,2.2,4.4,2.6,3.7,8.3,10.3,14.7,17.7,15.1,13.6,9.8,7.3,7.2,3.5,2.3,4.1,8.6,11.5,13.9,18.0,15.6,13.9,9.7,4.8,3.4,4.7,1.5,6.7,7.5,8.9,13.9,14.7,14.3,12.8,9.6,6.8,4.6,4.2,7.1,7.1,6.4,11.1,13.0,15.3,14.3,13.1,10.5,6.4,3.3,4.1,3.4,4.3,8.7,10.9,13.3,17.1,15.1,12.5,9.7,5.2,3.7,3.6,5.2,6.8,7.3,10.8,14.7,17.2,14.7,12.4,7.1,4.9,4.9,5.3,3.1,5.1,7.3,10.5,14.3,15.8,16.7,13.9,10.9,7.3,3.0,3.6,2.8,6.3,7.6,10.5,12.4,14.1,14.3,13.6,9.8,6.5,4.6,2.5,5.3,4.3,6.0,12.4,14.3,15.8,14.6,12.9,12.9,7.4,3.9,3.5,2.9,3.7,8.7,11.0,11.8,14.6,15.4,11.9,10.4,4.8,3.9,3.5,5.1,5.6,7.3,11.2,14.7,14.2,15.2,12.5,10.6,3.2,6.4,3.8,4.8,5.2,7.5,12.9,14.5,18.2,18.2,13.9,9.3,6.1,6.2,3.6,5.4,7.2,8.8,12.1,13.9,16.1,12.9,11.1,8.2,6.3,6.7,4.5,4.8,6.2,7.9,11.4,14.3,14.6,15.2,14.1,10.9,8.4,5.1,3.7,6.8,6.1,9.8,10.8,14.5,15.8,16.1,13.3,10.3,6.8,4.6,4.1,4.3,5.2,7.9,10.8,14.4,14.6,15.3,13.4,9.1,2.8,5.3,7.5,3.8,3.3,8.2,11.6,11.8,15.3,16.4,13.0,10.6,6.8,1.9,1.6,0.9,3.2,5.4,12.8,15.2,16.1,15.3,14.0,7.5,7.8,2.3,3.8,6.5,5.7,6.7,13.0,13.3,15.4,16.1,11.9,9.3,5.5,6.9,2.9,1.9,3.6,7.1,13.5,14.3,13.9,15.7,12.7,7.4,3.3,5.5,5.2,6.0,7.2,8.2,11.8,14.4,14.1,13.6,13.0,10.4,6.8,4.2,7.3,4.8,7.4,8.0,11.5,14.7,18.5,15.4,14.1,12.8,4.6,6.5,3.7,4.4,4.6,5.5,12.7,13.8,13.7,13.6,12.2,8.2,5.9,5.8,5.6,5.6,6.5,7.6,9.2,12.5,17.5,15.2,12.5,9.7,3.3,3.8,4.7,3.3,4.1,6.9,11.6,13.9,15.3,14.1,13.3,10.1,7.1,6.8,5.3,5.2,4.9,7.5,11.6,15.0,16.8,15.4,11.5,10.4,3.6,2.8,4.6,6.8,6.3,9.3,10.2,13.6,17.1,16.2,14.4,8.1,5.9,4.2,4.6,3.9,7.3,7.9,11.2,12.6,15.9,15.7,12.5,10.5,6.2,2.1,5.2,5.8,6.3,8.5,10.9,12.9,16.1,15.3,12.8,10.1,7.6,3.4,1.3,0.4,6.2,6.8,11.3,13.3,16.0,15.4,15.3,9.6,6.7,5.8,5.6,2.5,5.3,8.3,10.7,15.3,15.2,15.7,13.6,10.5,6.2,4.3,3.2,3.9,4.2,7.7,11.4,14.4,15.3,14.4,11.5,8.8,7.8,5.3,6.3,2.9,4.7,6.9,10.5,14.1,16.1,17.1,12.9,8.8,6.6,5.7,2.2,4.3,7.3,8.8,12.2,15.6,17.8,17.6,14.9,10.1,5.6,1.6,4.1,3.8,4.8,8.0,11.3,14.9,18.2,15.4,14.6,10.6,6.1,8.1,4.5,5.8,6.6,8.2,9.9,15.1,17.1,16.6,13.6,9.5,6.9,2.8,3.7,2.6,7.1,6.3,11.5,14.7,15.3,16.1,14.4,9.4,5.5,5.3,5.2,5.6,3.6,9.2,12.2,14.1,16.1,16.9,13.4,10.4,5.1,3.0,5.7,5.1,9.1,7.6,10.7,14.4,15.2,16.3,13.8,10.5,9.4,4.4,4.2,5.6,5.8,8.8,11.4,14.2,15.5,16.4,14.2,8.2,8.7,3.2,1.4,2.6,6.0,8.7,12.5,16.4,15.1,15.6,12.8,9.6,6.9,3.8,0.5,3.5,5.1,6.4,9.4,15.1,17.3,14.7,14.5,10.4,6.6,5.6,0.9,0.1,5.2,9.2,11.1,14.4,15.5,16.6,13.6,10.4,4.9,6.7,4.9,6.1,6.5,10.5,11.8,14.4,16.4,16.1,13.3,10.6,6.3,3.5,5.8,3.6,5.2,10.2,11.4,13.5,16.5,17.0,12.5,9.3,6.2,3.6,0.4,7.1,7.9,10.1,12.2,14.6,16.7,15.9,14.4,11.9,7.2,4.9,2.7,5.9,5.1,9.9,10.7,13.1,16.3,14.7,14.0,9.8,8.1,3.1,2.2,1.9,3.6,8.6,13.5,15.5,17.0,18.6,14.9,10.6,7.2,5.1,5.4,4.7,8.3,9.0,11.4,13.5,15.8,15.1,13.8,10.1,7.3,5.7,5.5,5.7,5.1,10.0,11.2,15.3,17.4,16.8,16.3,11.7,6.6,5.8,4.2,5.3,7.4,7.6,11.3,16.2,15.9,15.6,12.9,9.6,5.7,1.2,3.9,3.7,4.1,6.8,10.1,14.0,16.3,14.8,14.1,9.4,8.5,5.5,2.7,3.4,6.6,9.6,13.4,14.4,16.8,15.8,10.7,8.8,4.1,2.8,3.3,4.3,5.6,7.3,12.6,14.4,15.5,16.2,13.8,9.7,8.5,6.9,2.9,2.6,5.8,7.6,11.2,13.4,14.2,14.6,12.7,11.9,6.9,6.8,2.6,1.2,3.2,9.3,9.7,13.8,17.7,18.1,14.2,9.2,7.0,5.4,3.6,0.2,6.2,6.9,11.7,13.1,15.8,13.5,14.3,9.4,6.0,5.7,5.5,5.3,9.2,8.9,10.3,15.2,16.3,15.4,12.5,10.8,6.4,4.5,3.4,4.7,3.7,7.4,11.1,14.1,15.9,15.8,15.1,10.8,6.4,4.7,1.6,4.4,7.3,9.4,12.8,15.2,17.3,17.2,14.9,12.6,7.1,6.0,3.8,4.1,6.4,8.9,12.8,16.1,15.1,15.0,13.1,10.3,7.3,3.9,3.9,6.9,8.2,10.0,11.0,14.4,15.2,15.4,15.2,10.9,6.0,2.2,4.3,4.4,2.8,7.7,10.3,13.7,15.1,14.5,12.6,10.4,5.5,1.8,2.1,0.7,6.0,8.7,10.6,14.9,15.2,14.3,12.9,11.1,8.2,2.6,3.4,4.5,4.3,8.7,13.3,13.8,16.1,15.5,14.1,8.9,7.4,3.6,3.3,3.1,5.2,8.0,11.7,14.7,14.0,14.9,12.3,11.0,4.5,4.7,2.9,5.7,6.5,7.2,11.0,15.4,15.0,14.7,13.8,10.1,5.6,5.5,4.5,5.4,7.0,7.7,10.4,14.0,16.7,15.7,13.5,10.8,5.4,4.2,4.4,1.9,6.3,8.1,9.8,14.8,15.0,15.4,13.9,12.5,6.5,3.0,5.5,1.0,3.3,7.4,11.2,13.9,16.8,16.4,13.9,13.0,5.4,3.3,3.7,2.9,3.7,6.7,13.0,16.4,15.2,16.0,14.4,10.7,7.8,4.3];\n\n        // spot test against values taken from the reference implementation\n        let test = asap_smooth(&data, 100);\n        assert_eq!(test.len(), 93);\n        assert!((test[10] - 9.021034).abs() < 0.000001);\n        assert!((test[20] - 9.19).abs() < 0.000001);\n        assert!((test[30] - 9.068966).abs() < 0.000001);\n        assert!((test[40] - 9.237586).abs() < 0.000001);\n        assert!((test[50] - 9.145172).abs() < 0.000001);\n        assert!((test[60] - 9.014483).abs() < 0.000001);\n        assert!((test[70] - 9.293448).abs() < 0.000001);\n        assert!((test[80] - 9.417931).abs() < 0.000001);\n        assert!((test[90] - 9.602069).abs() < 0.000001);\n    }\n}\n"
  },
  {
    "path": "crates/count-min-sketch/Cargo.toml",
    "content": "[package]\nname = \"countminsketch\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nrand = \"0.8.4\"\nserde = { version = \"1.0\", features = [\"derive\"] }"
  },
  {
    "path": "crates/count-min-sketch/src/lib.rs",
    "content": "//! Count-Min Sketch implementation in Rust\n//!\n//! Based on the paper:\n//! <http://dimacs.rutgers.edu/~graham/pubs/papers/cm-full.pdf>\n\nuse std::{\n    fmt,\n    hash::{Hash, Hasher},\n};\n\n#[allow(deprecated)]\nuse std::hash::SipHasher;\n\nuse serde::{Deserialize, Serialize};\n\n/// The CountMinHashFn is a data structure used to hash items that are being\n/// added to a Count-Min Sketch.\n#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]\n#[repr(C)]\npub struct CountMinHashFn {\n    key: u64,\n}\n\nconst SEED: u64 = 0x517cc1b727220a95; // from FxHash\n\nimpl CountMinHashFn {\n    /// Creates a new CountMinHashFn whose hash function key is equal to `key`.\n    pub fn with_key(key: u64) -> Self {\n        Self { key }\n    }\n\n    /// Computes the hash of `item` according to the hash function and returns\n    /// the bucket index corresponding to the hashed value.\n    ///\n    /// The returned value will be between 0 and (`nbuckets` - 1).\n    #[allow(deprecated)]\n    pub fn hash_into_buckets<T: Hash>(&self, item: &T, nbuckets: usize) -> usize {\n        let (key1, key2) = (self.key, SEED);\n        let mut hasher = SipHasher::new_with_keys(key1, key2);\n        item.hash(&mut hasher);\n        let hash_val = hasher.finish();\n        (hash_val % (nbuckets as u64)) as usize\n    }\n\n    /// Returns the key for the hash function.\n    pub(crate) fn key(&self) -> u64 {\n        self.key\n    }\n}\n\n/// The Count-Min Sketch is a compact summary data structure capable of\n/// representing a high-dimensional vector and answering queries on this vector,\n/// in particular point queries and dot product queries, with strong accuracy\n/// guarantees. Such queries are at the core of many computations, so the\n/// structure can be used in order to answer a variety of other queries, such as\n/// frequent items (heavy hitters), quantile finding, join size estimation, and\n/// more. Since the data structure can easily process updates in the form of\n/// additions or subtractions to dimensions of the vector (which may correspond\n/// to insertions or deletions, or other transactions), it is capable of working\n/// over streams of updates, at high rates.[1]\n///\n/// [1]: <http://dimacs.rutgers.edu/~graham/pubs/papers/cmencyc.pdf>\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct CountMinSketch {\n    width: usize,\n    depth: usize,\n    // hashfuncs must be at least `depth` in length\n    hashfuncs: Vec<CountMinHashFn>,\n    // The outer and inner `Vec`s must be `depth` and `width` long, respectively\n    counters: Vec<Vec<i64>>,\n}\n\nimpl CountMinSketch {\n    /// Constructs a new Count-Min Sketch with the specified dimensions, using\n    /// `hashfuncs` to construct the underlying hash functions and `counters` to\n    /// populate the sketch with any data.\n    pub fn new(\n        width: usize,\n        depth: usize,\n        hashfuncs: Vec<CountMinHashFn>,\n        counters: Vec<Vec<i64>>,\n    ) -> Self {\n        assert_eq!(hashfuncs.len(), depth);\n        assert_eq!(counters.len(), depth);\n        assert_eq!(counters[0].len(), width);\n        Self {\n            width,\n            depth,\n            hashfuncs,\n            counters,\n        }\n    }\n\n    /// Constructs a new, empty Count-Min Sketch with the specified dimensions,\n    /// using `keys` to seed the underlying hash functions.\n    pub fn with_dims_and_hashfn_keys(width: usize, depth: usize, keys: Vec<u64>) -> Self {\n        assert_eq!(keys.len(), depth);\n        Self {\n            width,\n            depth,\n            hashfuncs: keys\n                .iter()\n                .map(|key| CountMinHashFn::with_key(*key))\n                .collect(),\n            counters: vec![vec![0; width]; depth],\n        }\n    }\n\n    /// Constructs a new, empty Count-Min Sketch with the specified dimensions.\n    pub fn with_dim(width: usize, depth: usize) -> Self {\n        let keys = (1..=depth).map(|k| k as u64).collect::<Vec<_>>();\n        CountMinSketch::with_dims_and_hashfn_keys(width, depth, keys)\n    }\n\n    /// Constructs a new, empty Count-Min Sketch whose dimensions will be\n    /// derived from the parameters.\n    ///\n    /// Then for any element *i*, an estimate of its count, âᵢ, will have the\n    /// guarantee:\n    ///         aᵢ ≤ âᵢ ≤ aᵢ + ϵN    with probability 1-δ\n    /// where aᵢ is the true count of element *i*\n    ///\n    /// Thus `epsilon` controls the error of the estimated count, relative to\n    /// the total number of items seen, and `delta` determines the probability\n    /// that the estimate will exceed the true count beyond the epsilon error\n    /// term.\n    ///\n    /// To accommodate this result, the sketch will have a width of ⌈e/ε⌉ and a\n    /// depth of ⌈ln(1/δ)⌉.\n    pub fn with_prob(epsilon: f64, delta: f64) -> Self {\n        assert!(0.0 < epsilon && epsilon < 1.0);\n        assert!(0.0 < delta && delta < 1.0);\n        let width = (1f64.exp() / epsilon).ceil() as usize;\n        let depth = (1f64 / delta).ln().ceil() as usize;\n        CountMinSketch::with_dim(width, depth)\n    }\n\n    /// Returns the width of the sketch.\n    pub fn width(&self) -> usize {\n        self.width\n    }\n\n    /// Returns the depth of the sketch.\n    pub fn depth(&self) -> usize {\n        self.depth\n    }\n\n    /// Returns a vector containing the keys of the hash functions used with the\n    /// sketch.\n    pub fn hash_keys(&self) -> Vec<u64> {\n        self.hashfuncs.iter().map(|f| f.key()).collect()\n    }\n\n    /// Returns a nested vector representing the sketch's counter table. Each\n    /// element in the outer vector corresponds to a row of the counter table,\n    /// and each element of the inner vector corresponds to the tally in that\n    /// bucket for a given row.\n    pub fn counters(&self) -> &Vec<Vec<i64>> {\n        &self.counters\n    }\n\n    /// Returns an estimate of the number of times `item` has been seen by the\n    /// sketch.\n    pub fn estimate<T: Hash>(&self, item: T) -> i64 {\n        let buckets = self\n            .hashfuncs\n            .iter()\n            .map(|h| h.hash_into_buckets(&item, self.width));\n\n        self.counters\n            .iter()\n            .zip(buckets)\n            .map(|(counter, bucket)| counter[bucket])\n            .min()\n            .unwrap()\n    }\n\n    /// Returns a vector of the indices for the buckets into which `item` hashes.\n    ///\n    /// The vector will have `self.depth` elements, each in the range\n    /// [0, self.width-1].\n    pub fn get_bucket_indices<T: Hash>(&self, item: T) -> Vec<usize> {\n        self.hashfuncs\n            .iter()\n            .map(|h| h.hash_into_buckets(&item, self.width))\n            .collect()\n    }\n\n    /// Adds the given `item` to the sketch.\n    pub fn add_value<T: Hash>(&mut self, item: T) {\n        for i in 0..self.depth {\n            let bucket = self.hashfuncs[i].hash_into_buckets(&item, self.width);\n            self.counters[i][bucket] += 1;\n        }\n    }\n\n    /// Subtract the given `item` from the sketch.\n    pub fn subtract_value<T: Hash>(&mut self, item: T) {\n        for i in 0..self.depth {\n            let bucket = self.hashfuncs[i].hash_into_buckets(&item, self.width);\n            self.counters[i][bucket] -= 1;\n        }\n    }\n\n    /// Includes the counts from `other` into `self` via elementwise addition of\n    /// the counter vectors.\n    ///\n    /// The underlying `CountMinHashFn`s in each sketch must have the same keys.\n    pub fn combine(&mut self, other: CountMinSketch) {\n        assert_eq!(self.width, other.width);\n        assert_eq!(self.depth, other.depth);\n        assert_eq!(self.hashfuncs, other.hashfuncs);\n        for (counter1, counter2) in self.counters.iter_mut().zip(other.counters) {\n            for (val1, val2) in counter1.iter_mut().zip(counter2) {\n                *val1 += val2;\n            }\n        }\n    }\n}\n\nimpl fmt::Display for CountMinSketch {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        writeln!(f, \"Count-Min Sketch:\")?;\n        write!(f, \"+------++\")?;\n        for _ in 0..self.width {\n            write!(f, \"--------+\")?;\n        }\n        writeln!(f)?;\n\n        write!(f, \"|      ||\")?;\n        for b in 0..self.width {\n            write!(f, \"    {b:>3} |\")?;\n        }\n        writeln!(f)?;\n\n        write!(f, \"+======++\")?;\n        for _ in 0..self.width {\n            write!(f, \"========+\")?;\n        }\n        writeln!(f)?;\n\n        for n in 0..self.depth {\n            write!(f, \"|  {n:>3} ||\")?;\n            for x in &self.counters[n] {\n                write!(f, \" {x:>6} |\")?;\n            }\n            writeln!(f)?;\n        }\n\n        write!(f, \"+------++\")?;\n        for _ in 0..self.width {\n            write!(f, \"--------+\")?;\n        }\n        writeln!(f)\n    }\n}\n"
  },
  {
    "path": "crates/count-min-sketch/tests/lib.rs",
    "content": "use countminsketch::CountMinSketch;\n\n#[test]\nfn empty_sketch() {\n    let cms = CountMinSketch::with_dim(1, 1);\n    assert_eq!(cms.estimate(\"foo\"), 0);\n}\n\n#[test]\nfn add_once() {\n    let mut cms = CountMinSketch::with_dim(2, 2);\n    cms.add_value(\"foo\");\n    assert_eq!(cms.estimate(\"foo\"), 1);\n}\n\n#[test]\nfn subtract_is_inverse_of_add() {\n    let mut cms = CountMinSketch::with_dim(2, 2);\n    cms.add_value(\"foo\");\n    cms.subtract_value(\"foo\");\n    assert_eq!(cms.estimate(\"foo\"), 0);\n}\n\n#[test]\nfn add_repeated() {\n    let mut cms = CountMinSketch::with_dim(2, 2);\n    for _ in 0..100_000 {\n        cms.add_value(\"foo\");\n    }\n    assert_eq!(cms.estimate(\"foo\"), 100_000);\n}\n\n#[test]\nfn add_repeated_with_collisions() {\n    // if sketch has width = 2 and we add 3 items, then we\n    // are guaranteed that we will have at least one hash\n    // collision in every row\n    let mut cms = CountMinSketch::with_dim(2, 5);\n\n    for _ in 0..100_000 {\n        cms.add_value(\"foo\")\n    }\n\n    for _ in 0..1_000 {\n        cms.add_value(\"bar\")\n    }\n\n    for _ in 0..1_000_000 {\n        cms.add_value(\"baz\")\n    }\n\n    let foo_est = cms.estimate(\"foo\");\n    let bar_est = cms.estimate(\"bar\");\n    let baz_est = cms.estimate(\"baz\");\n\n    let err_margin = (0.01 * (100_000f64 + 1_000f64 + 1_000_000f64)) as i64;\n    assert!(100_000 <= foo_est && foo_est < (100_000 + err_margin));\n    assert!(1_000 <= bar_est && bar_est < (1_000 + err_margin));\n    assert!(1_000_000 <= baz_est && baz_est < (1_000_000 + err_margin));\n}\n"
  },
  {
    "path": "crates/counter-agg/Cargo.toml",
    "content": "[package]\nname = \"counter-agg\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nflat_serialize = {path=\"../flat_serialize/flat_serialize\"}\nflat_serialize_macro = {path=\"../flat_serialize/flat_serialize_macro\"}\nserde = { version = \"1.0\", features = [\"derive\"] }\nstats_agg = {path=\"../stats-agg\"}\ntspoint = {path=\"../tspoint\"}\n\n[dev-dependencies]\napprox = \"0.5.1\""
  },
  {
    "path": "crates/counter-agg/src/lib.rs",
    "content": "use serde::{Deserialize, Serialize};\nuse stats_agg::{stats2d::StatsSummary2D, XYPair};\nuse std::fmt;\nuse tspoint::TSPoint;\n\npub mod range;\n\n#[cfg(test)]\nmod tests;\n\n#[derive(Debug, PartialEq, Eq)]\npub enum CounterError {\n    OrderError,\n    BoundsInvalid,\n}\n\n// TODO Intent is for this to be immutable with mutations going through (and\n//  internal consistency protected by) the builders below.  But, we allow raw\n//  access to the extension to allow it to (de)serialize, so the separation is\n//  but a fiction for now.  If the only consequence of corruption is\n//  nonsensical results rather than unsound behavior, garbage in garbage out.\n//  But much better if we can validate at deserialization.  We can do that in\n//  the builder if we want.\n#[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]\npub struct MetricSummary {\n    // TODO invariants?\n    pub first: TSPoint,\n    pub second: TSPoint,\n    pub penultimate: TSPoint,\n    pub last: TSPoint,\n    // Invariants:\n    // - num_changes > 0 if num_resets > 0\n    // - num_resets > 0 if num_changes > 0\n    // - reset_sum > 0 if num_resets > 0\n    // - num_resets > 0 if reset_sum > 0\n    pub reset_sum: f64,\n    pub num_resets: u64,\n    pub num_changes: u64,\n    // TODO Protect from deserialization?  Is there any risk other than giving\n    //  nonsensical results?  If so, maybe it's fine to just accept garbage\n    //  out upon garbage in.\n    pub stats: StatsSummary2D<f64>,\n    // TODO See TODOs in I64Range about protecting from deserialization.\n    pub bounds: Option<range::I64Range>,\n}\n\n// Note that this can lose fidelity with the timestamp, but it would only lose it in the microseconds,\n// this is likely okay in most applications. However, if you need better regression analysis at the subsecond level,\n// you can always subtract a common near value from all your times, then add it back in, the regression analysis will be unchanged.\n// Note that convert the timestamp into seconds rather than microseconds here so that the slope and any other regression analysis, is done on a per-second basis.\n// For instance, the slope will be the per-second slope, not the per-microsecond slope. The x intercept value will need to be converted back to microseconds so you get a timestamp out.\nfn ts_to_xy(pt: TSPoint) -> XYPair<f64> {\n    XYPair {\n        x: to_seconds(pt.ts as f64),\n        y: pt.val,\n    }\n}\n\nfn to_seconds(t: f64) -> f64 {\n    t / 1_000_000_f64 // by default postgres timestamps have microsecond precision\n}\n\n/// MetricSummary tracks monotonically increasing counters that may reset, ie every time the value decreases\n/// it is treated as a reset of the counter and the previous value is added to the \"true value\" of the\n/// counter at that timestamp.\nimpl MetricSummary {\n    pub fn new(pt: &TSPoint, bounds: Option<range::I64Range>) -> MetricSummary {\n        let mut n = MetricSummary {\n            first: *pt,\n            second: *pt,\n            penultimate: *pt,\n            last: *pt,\n            reset_sum: 0.0,\n            num_resets: 0,\n            num_changes: 0,\n            stats: StatsSummary2D::new(),\n            bounds,\n        };\n        n.stats.accum(ts_to_xy(*pt)).unwrap();\n        n\n    }\n\n    fn reset(&mut self, incoming: &TSPoint) {\n        if incoming.val < self.last.val {\n            self.reset_sum += self.last.val;\n            self.num_resets += 1;\n        }\n    }\n\n    // expects time-ordered input\n    fn add_point(&mut self, incoming: &TSPoint) -> Result<(), CounterError> {\n        if incoming.ts < self.last.ts {\n            return Err(CounterError::OrderError);\n        }\n        //TODO: test this\n        if incoming.ts == self.last.ts {\n            // if two points are equal we only use the first we see\n            // see discussion at https://github.com/timescale/timescaledb-toolkit/discussions/65\n            return Ok(());\n        }\n        // right now we treat a counter reset that goes to exactly zero as a change (not sure that's correct, but it seems defensible)\n        // These values are not rounded, so direct comparison is valid.\n        if incoming.val != self.last.val {\n            self.num_changes += 1;\n        }\n        if self.first == self.second {\n            self.second = *incoming;\n        }\n        self.penultimate = self.last;\n        self.last = *incoming;\n        let mut incoming_xy = ts_to_xy(*incoming);\n        incoming_xy.y += self.reset_sum;\n        self.stats.accum(incoming_xy).unwrap();\n        Ok(())\n    }\n\n    fn single_value(&self) -> bool {\n        self.last == self.first\n    }\n\n    // combining can only happen for disjoint time ranges\n    fn combine(&mut self, incoming: &MetricSummary) -> Result<(), CounterError> {\n        // this requires that self comes before incoming in time order\n        if self.last.ts >= incoming.first.ts {\n            return Err(CounterError::OrderError);\n        }\n\n        // These values are not rounded, so direct comparison is valid.\n        if self.last.val != incoming.first.val {\n            self.num_changes += 1;\n        }\n\n        if incoming.single_value() {\n            self.penultimate = self.last;\n        } else {\n            self.penultimate = incoming.penultimate;\n        }\n        if self.single_value() {\n            self.second = incoming.first;\n        }\n        let mut stats = incoming.stats;\n        // have to offset based on our reset_sum, including the amount we added based on any resets that happened at the boundary (but before we add in the incoming reset_sum)\n        stats\n            .offset(XYPair {\n                x: 0.0,\n                y: self.reset_sum,\n            })\n            .unwrap();\n        self.last = incoming.last;\n        self.reset_sum += incoming.reset_sum;\n        self.num_resets += incoming.num_resets;\n        self.num_changes += incoming.num_changes;\n\n        self.stats = self.stats.combine(stats).unwrap();\n        self.bounds_extend(incoming.bounds);\n        Ok(())\n    }\n\n    pub fn time_delta(&self) -> f64 {\n        to_seconds((self.last.ts - self.first.ts) as f64)\n    }\n\n    pub fn delta(&self) -> f64 {\n        self.last.val + self.reset_sum - self.first.val\n    }\n\n    pub fn rate(&self) -> Option<f64> {\n        if self.single_value() {\n            return None;\n        }\n        Some(self.delta() / self.time_delta())\n    }\n\n    pub fn idelta_left(&self) -> f64 {\n        //check for counter reset\n        if self.second.val >= self.first.val {\n            self.second.val - self.first.val\n        } else {\n            self.second.val // counter reset assumes it reset at the previous point, so we just return the second point\n        }\n    }\n\n    pub fn idelta_right(&self) -> f64 {\n        //check for counter reset\n        if self.last.val >= self.penultimate.val {\n            self.last.val - self.penultimate.val\n        } else {\n            self.last.val\n        }\n    }\n\n    pub fn irate_left(&self) -> Option<f64> {\n        if self.single_value() {\n            None\n        } else {\n            Some(self.idelta_left() / to_seconds((self.second.ts - self.first.ts) as f64))\n        }\n    }\n\n    pub fn irate_right(&self) -> Option<f64> {\n        if self.single_value() {\n            None\n        } else {\n            Some(self.idelta_right() / to_seconds((self.last.ts - self.penultimate.ts) as f64))\n        }\n    }\n\n    pub fn bounds_valid(&self) -> bool {\n        match self.bounds {\n            None => true, // unbounded contains everything\n            Some(b) => b.contains(self.last.ts) && b.contains(self.first.ts),\n        }\n    }\n\n    fn bounds_extend(&mut self, in_bounds: Option<range::I64Range>) {\n        match (self.bounds, in_bounds) {\n            (None, _) => self.bounds = in_bounds,\n            (_, None) => {}\n            (Some(mut a), Some(b)) => {\n                a.extend(&b);\n                self.bounds = Some(a);\n            }\n        };\n    }\n\n    // based on:  https://github.com/timescale/promscale_extension/blob/d51a0958442f66cb78d38b584a10100f0d278298/src/lib.rs#L208,\n    // which is based on:     // https://github.com/prometheus/prometheus/blob/e5ffa8c9a08a5ee4185271c8c26051ddc1388b7a/promql/functions.go#L59\n    pub fn prometheus_delta(&self) -> Result<Option<f64>, CounterError> {\n        if self.bounds.is_none() || !self.bounds_valid() || self.bounds.unwrap().has_infinite() {\n            return Err(CounterError::BoundsInvalid);\n        }\n        //must have at least 2 values\n        if self.single_value() || self.bounds.unwrap().is_singleton() {\n            //technically, the is_singleton check is redundant, it's included for clarity (any singleton bound that is valid can only be one point)\n            return Ok(None);\n        }\n\n        let mut result_val = self.delta();\n\n        // all calculated durations in seconds in Prom implementation, so we'll do that here.\n        // we can unwrap all of the bounds accesses as they are guaranteed to be there from the checks above\n        let mut duration_to_start =\n            to_seconds((self.first.ts - self.bounds.unwrap().left.unwrap()) as f64);\n\n        /* bounds stores [L,H), but Prom takes the duration using the inclusive range [L, H-1ms]. Subtract an extra ms, ours is in microseconds. */\n        let duration_to_end =\n            to_seconds((self.bounds.unwrap().right.unwrap() - self.last.ts - 1_000) as f64);\n        let sampled_interval = self.time_delta();\n        let avg_duration_between_samples = sampled_interval / (self.stats.n - 1) as f64; // don't have to worry about divide by zero because we know we have at least 2 values from the above.\n\n        // we don't want to extrapolate to negative counter values, so we calculate the duration to the zero point of the counter (based on what we know here) and set that as duration_to_start if it's smaller than duration_to_start\n        if result_val > 0.0 && self.first.val >= 0.0 {\n            let duration_to_zero = sampled_interval * (self.first.val / result_val);\n            if duration_to_zero < duration_to_start {\n                duration_to_start = duration_to_zero;\n            }\n        }\n\n        // If the first/last samples are close to the boundaries of the range,\n        // extrapolate the result. This is as we expect that another sample\n        // will exist given the spacing between samples we've seen thus far,\n        // with an allowance for noise.\n        // Otherwise, we extrapolate to one half the avg distance between samples...\n        // this was empirically shown to be good for certain things and was discussed at length in: https://github.com/prometheus/prometheus/pull/1161\n\n        let extrapolation_threshold = avg_duration_between_samples * 1.1;\n        let mut extrapolate_to_interval = sampled_interval;\n\n        if duration_to_start < extrapolation_threshold {\n            extrapolate_to_interval += duration_to_start\n        } else {\n            extrapolate_to_interval += avg_duration_between_samples / 2.0\n        }\n\n        if duration_to_end < extrapolation_threshold {\n            extrapolate_to_interval += duration_to_end\n        } else {\n            extrapolate_to_interval += avg_duration_between_samples / 2.0\n        }\n        result_val *= extrapolate_to_interval / sampled_interval;\n        Ok(Some(result_val))\n    }\n\n    pub fn prometheus_rate(&self) -> Result<Option<f64>, CounterError> {\n        let delta = self.prometheus_delta()?;\n        if delta.is_none() {\n            return Ok(None);\n        }\n        let delta = delta.unwrap();\n        let bounds = self.bounds.unwrap(); // if we got through delta without error then we have bounds\n        let duration = bounds.duration().unwrap() - 1_000; // bounds stores [L,H), but Prom takes the duration using the inclusive range [L, H-1ms]. So subtract an extra ms from the duration\n        if duration <= 0 {\n            return Ok(None); // if we have a total duration under a ms, it's less than prom could deal with so we return none.\n        }\n        Ok(Some(delta / to_seconds(duration as f64))) // don't have to deal with 0 case because that is checked in delta as well (singleton)\n    }\n}\n\nimpl fmt::Display for CounterError {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {\n        match self {\n            CounterError::OrderError => write!(\n                f,\n                \"out of order points: points must be submitted in time-order\"\n            ),\n            CounterError::BoundsInvalid => write!(f, \"cannot calculate delta without valid bounds\"),\n        }\n    }\n}\n\n#[derive(Clone, Debug, PartialEq, Deserialize, Serialize)]\npub struct GaugeSummaryBuilder(MetricSummary);\n\nimpl GaugeSummaryBuilder {\n    pub fn new(pt: &TSPoint, bounds: Option<range::I64Range>) -> Self {\n        Self(MetricSummary::new(pt, bounds))\n    }\n\n    /// expects time-ordered input\n    pub fn add_point(&mut self, incoming: &TSPoint) -> Result<(), CounterError> {\n        self.0.add_point(incoming)\n    }\n\n    /// combining can only happen for disjoint time ranges\n    pub fn combine(&mut self, incoming: &MetricSummary) -> Result<(), CounterError> {\n        self.0.combine(incoming)\n    }\n\n    pub fn set_bounds(&mut self, bounds: Option<range::I64Range>) {\n        self.0.bounds = bounds;\n    }\n\n    pub fn build(self) -> MetricSummary {\n        self.0\n    }\n\n    pub fn first(&self) -> &TSPoint {\n        &self.0.first\n    }\n\n    // TODO build method should check validity rather than caller\n    pub fn bounds_valid(&self) -> bool {\n        self.0.bounds_valid()\n    }\n}\n\nimpl From<MetricSummary> for GaugeSummaryBuilder {\n    fn from(summary: MetricSummary) -> Self {\n        Self(summary)\n    }\n}\n\n#[derive(Clone, Debug, PartialEq, Deserialize, Serialize)]\npub struct CounterSummaryBuilder(MetricSummary);\n\nimpl CounterSummaryBuilder {\n    pub fn new(pt: &TSPoint, bounds: Option<range::I64Range>) -> Self {\n        Self(MetricSummary::new(pt, bounds))\n    }\n\n    /// expects time-ordered input\n    pub fn add_point(&mut self, incoming: &TSPoint) -> Result<(), CounterError> {\n        self.0.reset(incoming);\n        self.0.add_point(incoming)\n    }\n\n    /// combining can only happen for disjoint time ranges\n    pub fn combine(&mut self, incoming: &MetricSummary) -> Result<(), CounterError> {\n        self.0.reset(&incoming.first);\n        self.0.combine(incoming)\n    }\n\n    pub fn set_bounds(&mut self, bounds: Option<range::I64Range>) {\n        self.0.bounds = bounds;\n    }\n\n    pub fn build(self) -> MetricSummary {\n        self.0\n    }\n\n    pub fn first(&self) -> &TSPoint {\n        &self.0.first\n    }\n\n    // TODO build method should check validity rather than caller\n    pub fn bounds_valid(&self) -> bool {\n        self.0.bounds_valid()\n    }\n}\n\nimpl From<MetricSummary> for CounterSummaryBuilder {\n    fn from(summary: MetricSummary) -> Self {\n        Self(summary)\n    }\n}\n"
  },
  {
    "path": "crates/counter-agg/src/range.rs",
    "content": "use serde::{Deserialize, Serialize};\nuse std::cmp::{max, min};\n\n// we always store ranges as half open, inclusive on left, exclusive on right,\n// we are a discrete type so translating is simple [), this enforces equality\n// between ranges like [0, 10) and [0, 9]\n// None values denote infinite bounds on that side\n#[derive(Debug, PartialEq, Eq, Copy, Clone, Serialize, Deserialize)]\n#[repr(C)]\npub struct I64Range {\n    pub left: Option<i64>,\n    pub right: Option<i64>,\n}\n\nimpl I64Range {\n    pub fn has_infinite(&self) -> bool {\n        self.left.is_none() || self.right.is_none()\n    }\n\n    // TODO See TODO below about range validity.  Right now we don't care\n    //  much.  If we start to care, move the caring to `new` and `extend`\n    //  methods.  That will allow this crate to protect the integrity of\n    //  MetricSummary and I64Range in the face of the extension needing to be\n    //  able to construct them from raw (and therefore potentially\n    //  corrupt) inputs.\n    fn is_valid(&self) -> bool {\n        match (self.left, self.right) {\n            (Some(a), Some(b)) => a <= b,\n            _ => true,\n        }\n    }\n\n    pub fn is_singleton(&self) -> bool {\n        match (self.left, self.right) {\n            (Some(a), Some(b)) => a == b,\n            _ => false,\n        }\n    }\n\n    pub fn extend(&mut self, other: &Self) {\n        // TODO: What should extend do with invalid ranges on either side? right now it treats them as if they are real...\n        self.left = match (self.left, other.left) {\n            (None, _) => None,\n            (_, None) => None,\n            (Some(a), Some(b)) => Some(min(a, b)),\n        };\n        self.right = match (self.right, other.right) {\n            (None, _) => None,\n            (_, None) => None,\n            (Some(a), Some(b)) => Some(max(a, b)),\n        };\n    }\n\n    pub fn contains(&self, pt: i64) -> bool {\n        match (self.left, self.right) {\n            (Some(l), Some(r)) => pt >= l && pt < r,\n            (Some(l), None) => pt >= l,\n            (None, Some(r)) => pt < r,\n            (None, None) => true,\n        }\n    }\n\n    // pub fn contains(&self, other: I64Range) -> bool{\n    //     unimplemented!()\n    // }\n    pub fn duration(&self) -> Option<i64> {\n        if self.has_infinite() || !self.is_valid() {\n            return None;\n        }\n        Some(self.right.unwrap() - self.left.unwrap())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    #[test]\n    fn test_extend() {\n        let mut a = I64Range {\n            left: Some(4),\n            right: Some(5),\n        };\n        let b = I64Range {\n            left: Some(3),\n            right: Some(6),\n        };\n        a.extend(&b);\n        // b completely covers a\n        assert_eq!(a, b);\n        // extend to left\n        let c = I64Range {\n            left: Some(2),\n            right: Some(5),\n        };\n        a.extend(&c);\n        assert_eq!(\n            a,\n            I64Range {\n                left: Some(2),\n                right: Some(6)\n            }\n        );\n        // extend to right\n        let d = I64Range {\n            left: Some(6),\n            right: Some(9),\n        };\n        a.extend(&d);\n        assert_eq!(\n            a,\n            I64Range {\n                left: Some(2),\n                right: Some(9)\n            }\n        );\n        // infinites\n        let e = I64Range {\n            left: Some(10),\n            right: None,\n        };\n        a.extend(&e);\n        assert_eq!(\n            a,\n            I64Range {\n                left: Some(2),\n                right: None\n            }\n        );\n        let f = I64Range {\n            left: None,\n            right: Some(5),\n        };\n        a.extend(&f);\n        assert_eq!(\n            a,\n            I64Range {\n                left: None,\n                right: None\n            }\n        );\n        // if a range contains another, it's unaffected\n        a.extend(&c);\n        assert_eq!(\n            a,\n            I64Range {\n                left: None,\n                right: None\n            }\n        );\n        // whether infinite or not\n        let mut a = I64Range {\n            left: Some(2),\n            right: Some(9),\n        };\n        a.extend(&b);\n        assert_eq!(\n            a,\n            I64Range {\n                left: Some(2),\n                right: Some(9)\n            }\n        );\n\n        // right now invalid ranges are are extended as normal though they can only ever extend in a single direction\n        let weird = I64Range {\n            left: Some(-2),\n            right: Some(-9),\n        };\n        a.extend(&weird);\n        assert_eq!(\n            a,\n            I64Range {\n                left: Some(-2),\n                right: Some(9)\n            }\n        );\n        let weird = I64Range {\n            left: Some(20),\n            right: Some(10),\n        };\n        a.extend(&weird);\n        assert_eq!(\n            a,\n            I64Range {\n                left: Some(-2),\n                right: Some(10)\n            }\n        );\n\n        //same if we extend a weird one, we can make a valid, or invalid one...\n        let mut weird = I64Range {\n            left: Some(-2),\n            right: Some(-9),\n        };\n\n        let weird2 = I64Range {\n            left: Some(-6),\n            right: Some(-10),\n        };\n        weird.extend(&weird2);\n        assert_eq!(\n            weird,\n            I64Range {\n                left: Some(-6),\n                right: Some(-9)\n            }\n        );\n        // it is also possible to get a valid range from two weirds\n        let weird3 = I64Range {\n            left: Some(6),\n            right: Some(3),\n        };\n        weird.extend(&weird3);\n        assert_eq!(\n            weird,\n            I64Range {\n                left: Some(-6),\n                right: Some(3)\n            }\n        );\n        assert!(weird.is_valid());\n\n        // extending with a valid should always produce a valid and will work as usual\n        let mut weird = I64Range {\n            left: Some(-6),\n            right: Some(-9),\n        };\n        let normal = I64Range {\n            left: Some(2),\n            right: Some(9),\n        };\n        weird.extend(&normal);\n        assert_eq!(\n            weird,\n            I64Range {\n                left: Some(-6),\n                right: Some(9)\n            }\n        );\n    }\n\n    #[test]\n    fn test_contains() {\n        let a = I64Range {\n            left: Some(2),\n            right: Some(5),\n        };\n        assert!(a.contains(2));\n        assert!(a.contains(4));\n        assert!(!a.contains(5));\n        assert!(!a.contains(6));\n\n        let a = I64Range {\n            left: None,\n            right: Some(-5),\n        };\n        assert!(a.contains(-100));\n        assert!(!a.contains(0));\n        assert!(!a.contains(6));\n\n        let a = I64Range {\n            left: Some(-10),\n            right: None,\n        };\n        assert!(a.contains(-10));\n        assert!(a.contains(0));\n        assert!(a.contains(1000));\n        assert!(!a.contains(-20));\n\n        //invalid ranges contain no points\n        let a = I64Range {\n            left: Some(0),\n            right: Some(-5),\n        };\n        assert!(!a.contains(-4));\n        assert!(!a.contains(1));\n        assert!(!a.contains(-6));\n    }\n\n    #[test]\n    fn test_duration() {\n        let a = I64Range {\n            left: Some(3),\n            right: Some(7),\n        };\n        assert_eq!(a.duration().unwrap(), 4);\n        let a = I64Range {\n            left: Some(-3),\n            right: Some(7),\n        };\n        assert_eq!(a.duration().unwrap(), 10);\n        let a = I64Range {\n            left: None,\n            right: Some(7),\n        };\n        assert_eq!(a.duration(), None);\n        let a = I64Range {\n            left: Some(3),\n            right: None,\n        };\n        assert_eq!(a.duration(), None);\n        //invalid ranges return None durations as well\n        let a = I64Range {\n            left: Some(3),\n            right: Some(0),\n        };\n        assert_eq!(a.duration(), None);\n    }\n\n    #[test]\n    fn test_checks() {\n        let a = I64Range {\n            left: Some(2),\n            right: Some(5),\n        };\n        assert!(a.is_valid());\n        assert!(!a.is_singleton());\n        let a = I64Range {\n            left: None,\n            right: Some(-5),\n        };\n        assert!(a.is_valid());\n        assert!(!a.is_singleton());\n        let a = I64Range {\n            left: Some(-10),\n            right: None,\n        };\n        assert!(a.is_valid());\n        assert!(!a.is_singleton());\n        let a = I64Range {\n            left: Some(2),\n            right: Some(2),\n        };\n        assert!(a.is_valid());\n        assert!(a.is_singleton());\n        assert_eq!(a.duration().unwrap(), 0);\n        let a = I64Range {\n            left: Some(0),\n            right: Some(-10),\n        };\n        assert!(!a.is_valid());\n        assert!(!a.is_singleton());\n    }\n}\n"
  },
  {
    "path": "crates/counter-agg/src/tests.rs",
    "content": "// TODO Move to ../tests/lib.rs\n\nuse crate::range::I64Range;\nuse crate::*;\nuse approx::assert_relative_eq;\nfn to_micro(t: f64) -> f64 {\n    t * 1_000_000.0\n}\n//do proper numerical comparisons on the values where that matters, use exact where it should be exact.\n#[track_caller]\npub fn assert_close_enough(p1: &MetricSummary, p2: &MetricSummary) {\n    assert_eq!(p1.first, p2.first, \"first\");\n    assert_eq!(p1.second, p2.second, \"second\");\n    assert_eq!(p1.penultimate, p2.penultimate, \"penultimate\");\n    assert_eq!(p1.last, p2.last, \"last\");\n    assert_eq!(p1.num_changes, p2.num_changes, \"num_changes\");\n    assert_eq!(p1.num_resets, p2.num_resets, \"num_resets\");\n    assert_eq!(p1.stats.n, p2.stats.n, \"n\");\n    assert_relative_eq!(p1.stats.sx, p2.stats.sx);\n    assert_relative_eq!(p1.stats.sx2, p2.stats.sx2);\n    assert_relative_eq!(p1.stats.sy, p2.stats.sy);\n    assert_relative_eq!(p1.stats.sy2, p2.stats.sy2);\n    assert_relative_eq!(p1.stats.sxy, p2.stats.sxy);\n}\n\n#[test]\nfn create() {\n    let testpt = TSPoint { ts: 0, val: 0.0 };\n    let test = CounterSummaryBuilder::new(&testpt, None).build();\n    assert_eq!(test.first, testpt);\n    assert_eq!(test.second, testpt);\n    assert_eq!(test.penultimate, testpt);\n    assert_eq!(test.last, testpt);\n    assert_eq!(test.reset_sum, 0.0);\n}\n#[test]\nfn adding_point() {\n    let mut test = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 0.0 }, None);\n    let testpt = TSPoint { ts: 5, val: 10.0 };\n\n    test.add_point(&testpt).unwrap();\n\n    let test = test.build();\n    assert_eq!(test.first, TSPoint { ts: 0, val: 0.0 });\n    assert_eq!(test.second, testpt);\n    assert_eq!(test.penultimate, TSPoint { ts: 0, val: 0.0 });\n    assert_eq!(test.last, testpt);\n    assert_eq!(test.reset_sum, 0.0);\n    assert_eq!(test.num_resets, 0);\n    assert_eq!(test.num_changes, 1);\n}\n\n#[test]\nfn adding_points_to_counter() {\n    let startpt = TSPoint { ts: 0, val: 0.0 };\n    let mut summary = CounterSummaryBuilder::new(&startpt, None);\n\n    summary.add_point(&TSPoint { ts: 5, val: 10.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 15, val: 20.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 20, val: 50.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 25, val: 10.0 }).unwrap();\n\n    let summary = summary.build();\n    assert_eq!(summary.first, startpt);\n    assert_eq!(summary.second, TSPoint { ts: 5, val: 10.0 });\n    assert_eq!(summary.penultimate, TSPoint { ts: 20, val: 50.0 });\n    assert_eq!(summary.last, TSPoint { ts: 25, val: 10.0 });\n    assert_relative_eq!(summary.reset_sum, 50.0);\n    assert_eq!(summary.num_resets, 1);\n    assert_eq!(summary.num_changes, 4);\n    assert_eq!(summary.stats.count(), 6);\n    assert_relative_eq!(summary.stats.sum().unwrap().x, 0.000075);\n    // non obvious one here, sumy should be the sum of all values including the resets at the time.\n    assert_relative_eq!(\n        summary.stats.sum().unwrap().y,\n        0.0 + 10.0 + 20.0 + 20.0 + 50.0 + 60.0\n    );\n}\n\n#[test]\nfn adding_out_of_order_counter() {\n    let startpt = TSPoint { ts: 0, val: 0.0 };\n    let mut summary = CounterSummaryBuilder::new(&startpt, None);\n\n    summary.add_point(&TSPoint { ts: 5, val: 10.0 }).unwrap();\n    assert_eq!(\n        CounterError::OrderError,\n        summary.add_point(&TSPoint { ts: 2, val: 9.0 }).unwrap_err()\n    );\n}\n\n#[test]\nfn test_counter_delta() {\n    let startpt = &TSPoint { ts: 0, val: 10.0 };\n    let mut summary = CounterSummaryBuilder::new(startpt, None);\n\n    // with one point\n    assert_relative_eq!(summary.clone().build().delta(), 0.0);\n\n    // simple case\n    summary.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n    assert_relative_eq!(summary.clone().build().delta(), 10.0);\n\n    //now with a reset\n    summary.add_point(&TSPoint { ts: 20, val: 10.0 }).unwrap();\n    assert_relative_eq!(summary.clone().build().delta(), 20.0);\n}\n\n#[test]\nfn test_combine() {\n    let mut summary = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 0.0 }, None);\n    summary.add_point(&TSPoint { ts: 5, val: 10.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 15, val: 30.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 20, val: 50.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 25, val: 10.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 30, val: 40.0 }).unwrap();\n\n    let mut part1 = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 0.0 }, None);\n    part1.add_point(&TSPoint { ts: 5, val: 10.0 }).unwrap();\n    part1.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n\n    let mut part2 = CounterSummaryBuilder::new(&TSPoint { ts: 15, val: 30.0 }, None);\n    part2.add_point(&TSPoint { ts: 20, val: 50.0 }).unwrap();\n    part2.add_point(&TSPoint { ts: 25, val: 10.0 }).unwrap();\n    part2.add_point(&TSPoint { ts: 30, val: 40.0 }).unwrap();\n\n    let mut combined = part1.clone();\n    combined.combine(&part2.clone().build()).unwrap();\n    assert_close_enough(&summary.build(), &combined.build());\n\n    // test error in wrong direction\n    assert_eq!(\n        part2.combine(&part1.build()).unwrap_err(),\n        CounterError::OrderError\n    );\n}\n\n#[test]\nfn test_combine_with_small_summary() {\n    let mut summary = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 50.0 }, None);\n    summary.add_point(&TSPoint { ts: 25, val: 10.0 }).unwrap();\n\n    // also tests that a reset at the boundary works correctly\n    let part1 = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 50.0 }, None);\n    let part2 = CounterSummaryBuilder::new(&TSPoint { ts: 25, val: 10.0 }, None);\n\n    let mut combined = part1.clone();\n    combined.combine(&part2.clone().build()).unwrap();\n    assert_close_enough(&summary.build(), &combined.build());\n\n    // test error in wrong direction\n    combined = part2;\n    assert_eq!(\n        combined.combine(&part1.build()).unwrap_err(),\n        CounterError::OrderError\n    );\n}\n#[test]\nfn test_multiple_resets() {\n    let startpt = TSPoint { ts: 0, val: 0.0 };\n    let mut summary = CounterSummaryBuilder::new(&startpt, None);\n\n    summary.add_point(&TSPoint { ts: 5, val: 10.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 15, val: 10.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 20, val: 40.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 25, val: 20.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 30, val: 40.0 }).unwrap();\n\n    let summary = summary.build();\n    assert_eq!(summary.first, startpt);\n    assert_eq!(summary.second, TSPoint { ts: 5, val: 10.0 });\n    assert_eq!(summary.penultimate, TSPoint { ts: 25, val: 20.0 });\n    assert_eq!(summary.last, TSPoint { ts: 30, val: 40.0 });\n    assert_relative_eq!(summary.reset_sum, 60.0);\n    assert_eq!(summary.num_resets, 2);\n    assert_eq!(summary.num_changes, 6);\n    assert_eq!(summary.stats.count(), 7);\n    assert_relative_eq!(summary.stats.sum().unwrap().x, 0.000105);\n    // non obvious one here, sy should be the sum of all values including the resets at the time they were added.\n    assert_relative_eq!(\n        summary.stats.sum().unwrap().y,\n        0.0 + 10.0 + 20.0 + 30.0 + 60.0 + 80.0 + 100.0\n    );\n\n    let mut part1 = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 0.0 }, None);\n    part1.add_point(&TSPoint { ts: 5, val: 10.0 }).unwrap();\n    part1.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n\n    let mut part2 = CounterSummaryBuilder::new(&TSPoint { ts: 15, val: 10.0 }, None);\n    part2.add_point(&TSPoint { ts: 20, val: 40.0 }).unwrap();\n    part2.add_point(&TSPoint { ts: 25, val: 20.0 }).unwrap();\n    part2.add_point(&TSPoint { ts: 30, val: 40.0 }).unwrap();\n\n    let mut combined = part1.clone();\n    combined.combine(&part2.clone().build()).unwrap();\n    assert_close_enough(&summary, &combined.build());\n\n    // test error in wrong direction\n    assert_eq!(\n        part2.combine(&part1.build()).unwrap_err(),\n        CounterError::OrderError\n    );\n}\n\n#[test]\nfn test_extraction_single_point() {\n    let startpt = TSPoint { ts: 20, val: 10.0 };\n    let summary = CounterSummaryBuilder::new(&startpt, None).build();\n    assert_relative_eq!(summary.delta(), 0.0);\n    assert_eq!(summary.rate(), None);\n    assert_relative_eq!(summary.idelta_left(), 0.0);\n    assert_relative_eq!(summary.idelta_right(), 0.0);\n    assert_eq!(summary.irate_left(), None);\n    assert_eq!(summary.irate_right(), None);\n    assert_eq!(summary.num_changes, 0);\n    assert_eq!(summary.num_resets, 0);\n}\n\n#[test]\nfn test_extraction_simple() {\n    let mut summary = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 0.0 }, None);\n    summary.add_point(&TSPoint { ts: 5, val: 5.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 10, val: 20.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 15, val: 30.0 }).unwrap();\n\n    let summary = summary.build();\n    assert_relative_eq!(summary.delta(), 30.0);\n    assert_relative_eq!(summary.rate().unwrap(), to_micro(2.0));\n    assert_relative_eq!(summary.idelta_left(), 5.0);\n    assert_relative_eq!(summary.idelta_right(), 10.0);\n    assert_relative_eq!(summary.irate_left().unwrap(), to_micro(1.0));\n    assert_relative_eq!(summary.irate_right().unwrap(), to_micro(2.0));\n    assert_eq!(summary.num_changes, 3);\n    assert_eq!(summary.num_resets, 0);\n}\n\n#[test]\nfn test_extraction_with_resets() {\n    let mut summary = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 10.0 }, None);\n    summary.add_point(&TSPoint { ts: 5, val: 5.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 10, val: 30.0 }).unwrap();\n    summary.add_point(&TSPoint { ts: 15, val: 15.0 }).unwrap();\n\n    let summary = summary.build();\n    assert_relative_eq!(summary.delta(), 45.0);\n    assert_relative_eq!(summary.rate().unwrap(), to_micro(3.0));\n    assert_relative_eq!(summary.idelta_left(), 5.0);\n    assert_relative_eq!(summary.idelta_right(), 15.0);\n    assert_relative_eq!(summary.irate_left().unwrap(), to_micro(1.0));\n    assert_relative_eq!(summary.irate_right().unwrap(), to_micro(3.0));\n    assert_eq!(summary.num_changes, 3);\n    assert_eq!(summary.num_resets, 2);\n}\n\n#[test]\nfn test_bounds() {\n    let summary = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 10.0 }, None);\n    assert!(summary.bounds_valid()); // no bound is fine.\n\n    let summary = CounterSummaryBuilder::new(\n        &TSPoint { ts: 0, val: 10.0 },\n        Some(I64Range {\n            left: Some(5),\n            right: Some(10),\n        }),\n    );\n    assert!(!summary.bounds_valid()); // wrong bound not\n\n    // left bound inclusive\n    let mut summary = CounterSummaryBuilder::new(\n        &TSPoint { ts: 0, val: 10.0 },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(10),\n        }),\n    );\n    assert!(summary.bounds_valid());\n    summary.add_point(&TSPoint { ts: 5, val: 5.0 }).unwrap();\n    assert!(summary.bounds_valid());\n\n    // adding points past our bounds is okay, but the bounds will be invalid when we check, this will happen in the final function not on every point addition for efficiency\n    // note the right bound is exclusive\n    summary.add_point(&TSPoint { ts: 10, val: 10.0 }).unwrap();\n    assert!(!summary.bounds_valid());\n\n    // slightly weird case here... two invalid bounds can produce a validly bounded object once the bounds are combined, this is a bit weird, but seems like it's the correct behavior\n    let summary2 = CounterSummaryBuilder::new(\n        &TSPoint { ts: 15, val: 10.0 },\n        Some(I64Range {\n            left: Some(20),\n            right: Some(30),\n        }),\n    );\n    summary.combine(&summary2.build()).unwrap();\n    assert!(summary.bounds_valid());\n    assert_eq!(\n        summary.clone().build().bounds.unwrap(),\n        I64Range {\n            left: Some(0),\n            right: Some(30)\n        }\n    );\n\n    // two of the same valid bounds remain the same and valid\n    let summary2 = CounterSummaryBuilder::new(\n        &TSPoint { ts: 20, val: 10.0 },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(30),\n        }),\n    );\n    summary.combine(&summary2.build()).unwrap();\n    assert!(summary.bounds_valid());\n    assert_eq!(\n        summary.clone().build().bounds.unwrap(),\n        I64Range {\n            left: Some(0),\n            right: Some(30)\n        }\n    );\n\n    // combining with unbounded ones is fine, but the bounds survive\n    let summary2 = CounterSummaryBuilder::new(&TSPoint { ts: 25, val: 10.0 }, None);\n    summary.combine(&summary2.build()).unwrap();\n    assert!(summary.bounds_valid());\n    assert_eq!(\n        summary.clone().build().bounds.unwrap(),\n        I64Range {\n            left: Some(0),\n            right: Some(30)\n        }\n    );\n\n    // and combining bounds that do not span are still invalid\n    let summary2 = CounterSummaryBuilder::new(\n        &TSPoint { ts: 35, val: 10.0 },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(32),\n        }),\n    );\n    summary.combine(&summary2.build()).unwrap();\n    assert!(!summary.bounds_valid());\n    assert_eq!(\n        summary.build().bounds.unwrap(),\n        I64Range {\n            left: Some(0),\n            right: Some(32)\n        }\n    );\n\n    // combining unbounded with bounded ones is fine, but the bounds survive\n    let mut summary = CounterSummaryBuilder::new(&TSPoint { ts: 0, val: 10.0 }, None);\n    let summary2 = CounterSummaryBuilder::new(\n        &TSPoint { ts: 25, val: 10.0 },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(30),\n        }),\n    );\n    summary.combine(&summary2.build()).unwrap();\n    assert!(summary.bounds_valid());\n    assert_eq!(\n        summary.build().bounds.unwrap(),\n        I64Range {\n            left: Some(0),\n            right: Some(30)\n        }\n    );\n}\n\n#[test]\nfn test_prometheus_extrapolation_simple() {\n    //error on lack of bounds provided\n    let summary = CounterSummaryBuilder::new(\n        &TSPoint {\n            ts: 5000,\n            val: 15.0,\n        },\n        None,\n    );\n    let summary = summary.build();\n    assert_eq!(\n        summary.prometheus_delta().unwrap_err(),\n        CounterError::BoundsInvalid\n    );\n    assert_eq!(\n        summary.prometheus_rate().unwrap_err(),\n        CounterError::BoundsInvalid\n    );\n\n    //error on infinite bounds\n    let summary = CounterSummaryBuilder::new(\n        &TSPoint {\n            ts: 5000,\n            val: 15.0,\n        },\n        Some(I64Range {\n            left: None,\n            right: Some(21000),\n        }),\n    )\n    .build();\n    assert_eq!(\n        summary.prometheus_delta().unwrap_err(),\n        CounterError::BoundsInvalid\n    );\n    assert_eq!(\n        summary.prometheus_rate().unwrap_err(),\n        CounterError::BoundsInvalid\n    );\n\n    //ranges less than 1ms are treated as zero by Prom\n    let mut summary = CounterSummaryBuilder::new(\n        &TSPoint { ts: 300, val: 15.0 },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(900),\n        }),\n    );\n    summary.add_point(&TSPoint { ts: 600, val: 20.0 }).unwrap();\n    assert_eq!(summary.build().prometheus_rate().unwrap(), None);\n\n    //ranges should go out an extra 1000 so that we account for the extra duration that prom subtracts (1 ms)\n    let mut summary = CounterSummaryBuilder::new(\n        &TSPoint {\n            ts: 5000,\n            val: 15.0,\n        },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(21000),\n        }),\n    );\n    // singletons should return none\n    assert_eq!(summary.clone().build().prometheus_delta().unwrap(), None);\n    assert_eq!(summary.clone().build().prometheus_rate().unwrap(), None);\n\n    // TODO Was this intentional?  add_point and then we immediately discard!\n    summary\n        .add_point(&TSPoint {\n            ts: 10000,\n            val: 20.0,\n        })\n        .unwrap();\n\n    //ranges should go out an extra 1000 so that we account for the extra duration that prom subtracts (1 ms)\n    let mut summary = CounterSummaryBuilder::new(\n        &TSPoint {\n            ts: 5000,\n            val: 15.0,\n        },\n        Some(I64Range {\n            left: Some(0),\n            right: Some(21000),\n        }),\n    );\n    // singletons should return none\n    assert_eq!(summary.clone().build().prometheus_delta().unwrap(), None);\n    assert_eq!(summary.clone().build().prometheus_rate().unwrap(), None);\n\n    summary\n        .add_point(&TSPoint {\n            ts: 10000,\n            val: 20.0,\n        })\n        .unwrap();\n    summary\n        .add_point(&TSPoint {\n            ts: 15000,\n            val: 25.0,\n        })\n        .unwrap();\n\n    let summary = summary.build();\n    assert_relative_eq!(summary.delta(), 10.0);\n    assert_relative_eq!(summary.rate().unwrap(), to_micro(0.001));\n    assert_relative_eq!(summary.prometheus_delta().unwrap().unwrap(), 20.0);\n    // linear cases like this should be equal\n    assert_relative_eq!(\n        summary.prometheus_rate().unwrap().unwrap(),\n        summary.rate().unwrap()\n    );\n\n    // add a point outside our bounds and make sure we error correctly\n    let mut summary = CounterSummaryBuilder::from(summary);\n    summary\n        .add_point(&TSPoint {\n            ts: 25000,\n            val: 35.0,\n        })\n        .unwrap();\n    let summary = summary.build();\n    assert_eq!(\n        summary.prometheus_delta().unwrap_err(),\n        CounterError::BoundsInvalid\n    );\n    assert_eq!(\n        summary.prometheus_rate().unwrap_err(),\n        CounterError::BoundsInvalid\n    );\n}\n\n#[test]\nfn test_prometheus_extrapolation_bound_size() {\n    let mut summary = CounterSummaryBuilder::new(\n        &TSPoint {\n            ts: 20000,\n            val: 40.0,\n        },\n        Some(I64Range {\n            left: Some(10000),\n            right: Some(51000),\n        }),\n    );\n    summary\n        .add_point(&TSPoint {\n            ts: 30000,\n            val: 20.0,\n        })\n        .unwrap();\n    summary\n        .add_point(&TSPoint {\n            ts: 40000,\n            val: 40.0,\n        })\n        .unwrap();\n    let summary = summary.build();\n    assert_relative_eq!(summary.delta(), 40.0);\n    assert_relative_eq!(summary.rate().unwrap(), to_micro(0.002));\n    //we go all the way to the edge of the bounds here because it's within 1.1 average steps (when you subtract the extra 1000 for ms it goes to 50000)\n    assert_relative_eq!(summary.prometheus_delta().unwrap().unwrap(), 80.0);\n    // linear cases like this should be equal\n    assert_relative_eq!(\n        summary.prometheus_rate().unwrap().unwrap(),\n        summary.rate().unwrap()\n    );\n\n    // now lets push the bounds to be a bit bigger\n    let mut summary = CounterSummaryBuilder::from(summary);\n    summary.set_bounds(Some(I64Range {\n        left: Some(8000),\n        right: Some(53000),\n    }));\n    // now because we're further than 1.1 out on each side, we end projecting out to half the avg distance on each side\n    assert_relative_eq!(\n        summary.clone().build().prometheus_delta().unwrap().unwrap(),\n        60.0\n    );\n    // but the rate is still divided by the full bound duration\n    assert_relative_eq!(\n        summary.build().prometheus_rate().unwrap().unwrap(),\n        to_micro(60.0 / 44000.0)\n    );\n\n    //this should all be the same as the last one in the first part.\n    // The change occurs because we hit the zero boundary condition\n    // so things change on the second bit because of where resets occur and our starting value\n    let mut summary = CounterSummaryBuilder::new(\n        &TSPoint {\n            ts: 20000,\n            val: 20.0,\n        },\n        Some(I64Range {\n            left: Some(10000),\n            right: Some(51000),\n        }),\n    );\n    summary\n        .add_point(&TSPoint {\n            ts: 30000,\n            val: 40.0,\n        })\n        .unwrap();\n    summary\n        .add_point(&TSPoint {\n            ts: 40000,\n            val: 20.0,\n        })\n        .unwrap();\n    let summary = summary.build();\n    assert_relative_eq!(summary.delta(), 40.0);\n    assert_relative_eq!(summary.rate().unwrap(), to_micro(0.002));\n    //we go all the way to the edge of the bounds here because it's within 1.1 average steps\n    assert_relative_eq!(summary.prometheus_delta().unwrap().unwrap(), 80.0);\n    // linear cases like this should be equal\n    assert_relative_eq!(\n        summary.prometheus_rate().unwrap().unwrap(),\n        summary.rate().unwrap()\n    );\n\n    // now lets push the bounds to be a bit bigger\n    let mut summary = CounterSummaryBuilder::from(summary);\n    summary.set_bounds(Some(I64Range {\n        left: Some(8000),\n        right: Some(53000),\n    }));\n    let summary = summary.build();\n    // now because we're further than 1.1 out on the right side,\n    // we end projecting out to half the avg distance on that side,\n    // but because we hit the inferred zero point  on the left (0 in this case)\n    // we use zero as the bound on the left side\n    assert_relative_eq!(summary.prometheus_delta().unwrap().unwrap(), 70.0);\n    // but the rate is still divided by the full bound duration\n    assert_relative_eq!(\n        summary.prometheus_rate().unwrap().unwrap(),\n        to_micro(70.0 / 44000.0)\n    );\n}\n"
  },
  {
    "path": "crates/encodings/Cargo.toml",
    "content": "[package]\nname = \"encodings\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\n\n[dev-dependencies]\nquickcheck = \"1\"\nquickcheck_macros = \"1\""
  },
  {
    "path": "crates/encodings/src/lib.rs",
    "content": "pub mod delta {\n    use crate::zigzag;\n\n    pub fn i64_decoder() -> impl FnMut(i64) -> i64 {\n        let mut prev = 0i64;\n        move |delta| {\n            let value = prev.wrapping_add(delta);\n            prev = value;\n            value\n        }\n    }\n\n    pub fn u64_decoder() -> impl FnMut(u64) -> u64 {\n        let mut prev = 0u64;\n        move |delta| {\n            let delta = zigzag::decode(delta) as u64;\n            let value = prev.wrapping_add(delta);\n            prev = value;\n            value\n        }\n    }\n\n    pub fn i64_encoder() -> impl FnMut(i64) -> i64 {\n        let mut prev = 0i64;\n        move |value: i64| {\n            let delta = value.wrapping_sub(prev);\n            prev = value;\n            delta\n        }\n    }\n\n    pub fn u64_encoder() -> impl FnMut(u64) -> u64 {\n        let mut prev = 0u64;\n        move |value: u64| {\n            let delta = value.wrapping_sub(prev);\n            prev = value;\n            zigzag::encode(delta as i64)\n        }\n    }\n\n    #[cfg(test)]\n    mod test {\n        use quickcheck_macros::quickcheck;\n\n        use super::*;\n\n        #[quickcheck]\n        fn quick_test_roundtrip_u64(values: Vec<u64>) -> bool {\n            let mut bytes = vec![];\n            crate::prefix_varint::compress_u64s_to_vec(\n                &mut bytes,\n                values.iter().cloned().map(u64_encoder()),\n            );\n\n            let output: Vec<u64> = crate::prefix_varint::u64_decompressor(&bytes)\n                .map(u64_decoder())\n                .collect();\n            assert_eq!(values, output);\n            true\n        }\n\n        #[quickcheck]\n        fn quick_test_roundtrip_i64(values: Vec<i64>) -> bool {\n            let mut bytes = vec![];\n            crate::prefix_varint::compress_i64s_to_vec(\n                &mut bytes,\n                values.iter().cloned().map(i64_encoder()),\n            );\n\n            let output: Vec<i64> = crate::prefix_varint::i64_decompressor(&bytes)\n                .map(i64_decoder())\n                .collect();\n            assert_eq!(values, output);\n            true\n        }\n    }\n}\n\npub mod zigzag {\n    #[inline(always)]\n    pub fn encode(n: i64) -> u64 {\n        if n < 0 {\n            // let's avoid the edge case of i64::min_value()\n            // !n is equal to `-n - 1`, so this is:\n            // !n * 2 + 1 = 2(-n - 1) + 1 = -2n - 2 + 1 = -2n - 1\n            !(n as u64) * 2 + 1\n        } else {\n            (n as u64) * 2\n        }\n    }\n\n    #[inline(always)]\n    pub fn decode(n: u64) -> i64 {\n        if n % 2 == 0 {\n            // positive number\n            (n / 2) as i64\n        } else {\n            // negative number\n            // !m * 2 + 1 = n\n            // !m * 2 = n - 1\n            // !m = (n - 1) / 2\n            // m = !((n - 1) / 2)\n            // since we have n is odd, we have floor(n / 2) = floor((n - 1) / 2)\n            !(n / 2) as i64\n        }\n    }\n}\n\npub mod prefix_varint {\n    //! Similar to [LEB128](https://en.wikipedia.org/wiki/LEB128), but it moves\n    //! all the tag bits to the LSBs of the first byte, which ends up looking\n    //! like this (`x` is a value bit, the rest are tag bits):\n    //! ```python,ignore,no_run\n    //! xxxxxxx1  7 bits in 1 byte\n    //! xxxxxx10 14 bits in 2 bytes\n    //! xxxxx100 21 bits in 3 bytes\n    //! xxxx1000 28 bits in 4 bytes\n    //! xxx10000 35 bits in 5 bytes\n    //! xx100000 42 bits in 6 bytes\n    //! x1000000 49 bits in 7 bytes\n    //! 10000000 56 bits in 8 bytes\n    //! 00000000 64 bits in 9 bytes\n    //! ```\n    //! based on https://github.com/stoklund/varint\n\n    pub fn size_vec<I: Iterator<Item = u64>>(bytes: &mut Vec<u8>, values: I) {\n        let size: usize = values.map(|v| bytes_for_value(v) as usize).sum();\n        bytes.reserve(size + 9);\n    }\n\n    #[inline]\n    pub fn bytes_for_value(value: u64) -> u32 {\n        let bits = value.leading_zeros();\n        let mut bytes = 1 + bits.wrapping_sub(1) / 7;\n        if bits > 56 {\n            bytes = 9\n        }\n        bytes\n    }\n    pub struct I64Compressor<F: FnMut(i64) -> i64> {\n        compressor: U64Compressor<fn(u64) -> u64>,\n        encoder: F,\n    }\n\n    impl I64Compressor<fn(i64) -> i64> {\n        pub fn new() -> Self {\n            Self {\n                compressor: U64Compressor::new(),\n                encoder: |i| i,\n            }\n        }\n    }\n\n    impl Default for I64Compressor<fn(i64) -> i64> {\n        fn default() -> Self {\n            Self::new()\n        }\n    }\n\n    impl<F: FnMut(i64) -> i64> I64Compressor<F> {\n        pub fn with(encoder: F) -> Self {\n            Self {\n                compressor: U64Compressor::new(),\n                encoder,\n            }\n        }\n\n        pub fn push(&mut self, value: i64) {\n            let encoded = crate::zigzag::encode((self.encoder)(value));\n            self.compressor.push(encoded)\n        }\n\n        pub fn finish(self) -> Vec<u8> {\n            self.compressor.finish()\n        }\n    }\n\n    pub struct U64Compressor<F: FnMut(u64) -> u64> {\n        bytes: Vec<u8>,\n        encoder: F,\n    }\n\n    impl U64Compressor<fn(u64) -> u64> {\n        pub fn new() -> Self {\n            Self {\n                bytes: vec![],\n                encoder: |i| i,\n            }\n        }\n    }\n\n    impl Default for U64Compressor<fn(u64) -> u64> {\n        fn default() -> Self {\n            Self::new()\n        }\n    }\n\n    impl<F: FnMut(u64) -> u64> U64Compressor<F> {\n        pub fn with(encoder: F) -> Self {\n            Self {\n                bytes: vec![],\n                encoder,\n            }\n        }\n\n        pub fn push(&mut self, value: u64) {\n            let encoded = (self.encoder)(value);\n            write_to_vec(&mut self.bytes, encoded);\n        }\n\n        pub fn finish(self) -> Vec<u8> {\n            self.bytes\n        }\n\n        pub fn is_empty(&self) -> bool {\n            self.bytes.is_empty()\n        }\n    }\n\n    pub fn compress_i64s_to_vec<I: Iterator<Item = i64>>(bytes: &mut Vec<u8>, values: I) {\n        compress_u64s_to_vec(bytes, values.map(crate::zigzag::encode))\n    }\n\n    pub fn compress_u64s_to_vec<I: Iterator<Item = u64>>(bytes: &mut Vec<u8>, values: I) {\n        values.for_each(|v| write_to_vec(bytes, v));\n    }\n\n    // based on https://github.com/stoklund/varint, (Apache licensed)\n    // see also https://github.com/WebAssembly/design/issues/601\n    #[inline]\n    pub fn write_to_vec(out: &mut Vec<u8>, mut value: u64) {\n        if value == 0 {\n            out.push(0x1);\n            return;\n        }\n        let bits = 64 - value.leading_zeros();\n        let mut bytes = 1 + bits.wrapping_sub(1) / 7;\n        if bits > 56 {\n            out.push(0);\n            bytes = 8\n        } else if value != 0 {\n            value = (2 * value + 1) << (bytes - 1)\n        }\n        let value = value.to_le_bytes();\n        for value in value.iter().take(bytes as usize) {\n            out.push(*value);\n        }\n    }\n\n    type Value = u64;\n\n    pub fn i64_decompressor(bytes: &[u8]) -> impl Iterator<Item = i64> + '_ {\n        u64_decompressor(bytes).map(crate::zigzag::decode)\n    }\n\n    pub fn u64_decompressor(mut bytes: &[u8]) -> impl Iterator<Item = u64> + '_ {\n        std::iter::from_fn(move || {\n            if bytes.is_empty() {\n                None\n            } else {\n                let (value, len) = read_from_slice(bytes);\n                bytes = &bytes[len..];\n                Some(value)\n            }\n        })\n    }\n\n    #[inline]\n    pub fn read_from_slice(bytes: &[u8]) -> (Value, usize) {\n        let value: [u8; 8] = if bytes.len() >= 8 {\n            bytes[0..8].try_into().unwrap()\n        } else {\n            let mut value = [0; 8];\n            value[..bytes.len()].copy_from_slice(bytes);\n            value\n        };\n        let tag_byte = value[0];\n        if tag_byte & 1 == 1 {\n            let value = (tag_byte >> 1) as u64;\n            return (value, 1);\n        }\n        let length = prefix_length(tag_byte) as usize;\n        let value = if length < 9 {\n            let unused = 64 - 8 * length;\n            let value = u64::from_le_bytes(value);\n            (value << unused) >> (unused + length)\n        } else {\n            u64::from_le_bytes(bytes[1..9].try_into().unwrap())\n        };\n\n        (value, length)\n    }\n\n    #[inline(always)]\n    pub fn prefix_length(tag_byte: u8) -> u32 {\n        1 + ((tag_byte as u32) | 0x100).trailing_zeros()\n    }\n\n    #[cfg(test)]\n    mod test {\n        use quickcheck_macros::quickcheck;\n\n        use super::*;\n\n        #[quickcheck]\n        fn quick_test_roundtrip_u64(values: Vec<u64>) -> bool {\n            let mut bytes = vec![];\n            compress_u64s_to_vec(&mut bytes, values.iter().cloned());\n\n            let output: Vec<u64> = u64_decompressor(&bytes).collect();\n            assert_eq!(values, output);\n            true\n        }\n\n        #[quickcheck]\n        fn quick_test_roundtrip_i64(values: Vec<i64>) -> bool {\n            let mut bytes = vec![];\n            compress_i64s_to_vec(&mut bytes, values.iter().cloned());\n\n            let output: Vec<i64> = i64_decompressor(&bytes).collect();\n            assert_eq!(values, output);\n            true\n        }\n    }\n}\n"
  },
  {
    "path": "crates/flat_serialize/Readme.md",
    "content": "# Flat Serialize #\n\nA cannonicalization of write-to-pointer style serialization. You write a\ndefinition describing the layout the data should have when serialized, and the\nmacro will generate code that reads and writes each field in order. It also\nsupports variable-length fields where the length is stored in an earlier field.\n\n## Examples ##\n\n### Basic ###\n\n```rust\n/// This will define a struct like\n/// ```\n/// struct Basic<'a> {\n///     header: u32,\n///     data_len: usize,\n///     array: [u16; 3],\n///     data: &'a [u8],\n///     data2: &'a [u8],\n/// }\n/// ```\n/// along with various functions to read and write this data to byte buffers\n/// (see below)\nflat_serialize!{\n    struct Basic<'a> {\n        header: u32,\n        data_len: usize,\n        array: [u16; 3],\n        data: [u8; self.data_len],\n        data2: [u8; self.data_len / 2],\n    }\n}\n\n\n#[test]\nfn basic() {\n    let basic = Basic{\n        header: 33,\n        array: [202, 404, 555],\n        data: &[1, 3, 5, 7, 9, 11],\n        data2: &[4, 4, 4],\n    };\n\n    // The generated struct can be used to serialize data to a byte vector\n    let &mut serialized = Vec::with_capacity(basic.len());\n    basic.fill_vec(&mut serialized)\n\n    // or deserialize data from a vector so written\n    let (deserialized, remaining_bytes) = unsafe {\n        Basic::try_ref(&bytes).unwrap()\n    };\n    assert_eq!(deserialized.header, &33);\n    assert_eq!(deserialized.array, &[202, 404, 555]);\n    assert_eq!(deserialized.data, &[1, 3, 5, 7, 9, 11][..]);\n    assert_eq!(deserialized.data2, &[4, 4, 4][..]);\n    assert_eq!(remaining_bytes, &[][..]);\n\n    // For serialization, the generated code will simply write each field, one\n    // after another. (It is currently the programmer's responsibility to ensure\n    // that the fields will be aligned correctly for deserialization)\n    let mut expected = Vec::new();\n    bytes.extend_from_slice(&33u32.to_ne_bytes());\n    bytes.extend_from_slice(&6usize.to_ne_bytes());\n    bytes.extend_from_slice(&202u16.to_ne_bytes());\n    bytes.extend_from_slice(&404u16.to_ne_bytes());\n    bytes.extend_from_slice(&555u16.to_ne_bytes());\n    bytes.extend_from_slice(&[1, 3, 5, 7, 9, 11]);\n    bytes.extend_from_slice(&[4, 4, 4]);\n    assert_eq!(serialized, expected);\n}\n```\n\n### Advanced ###\n\n```rust\n/// flat-serializable values can be nested, a field marked with\n/// `flat_serialize::flatten` will be read and written using `FlattenableRef`.\n//  The data layout is equivalent to just inlining all the fields.\n/// ```\n/// struct Nested<'a> {\n///     prefix: u64,\n///     basic: Basic<'a>,\n/// }\n/// ```\nflat_serialize!{\n    struct Nested<'a> {\n        prefix: u64,\n        #[flat_serialize::flatten]\n        basic: Basic<'a>,\n    }\n}\n\n/// Enum-like values are also supported. The enum tag is stored immediately\n/// before the enum fields.\nflat_serialize!{\n    enum Enum<'a> {\n        k: u64,\n        First: 2 {\n            data_len: u32,\n            data: [u8; self.data_len],\n        },\n        Fixed: 3 {\n            array: [u16; 3],\n        },\n    }\n}\n\nfn enum_example(e: Enum) {\n    match e {\n        Enum::First{ data_len, data } => todo!(),\n        Enum::Fixed{ array } => todo!(),\n    }\n}\n```\n"
  },
  {
    "path": "crates/flat_serialize/example_generated.rs",
    "content": "#![allow(unused_imports)]\nuse crate as flat_serialize;\n#[derive(Clone, Debug)]\npub struct Basic<'input> {\n    pub header: u64,\n    pub data_len: u32,\n    pub array: [u16; 3],\n    pub data: <u8 as flat_serialize::FlatSerializable<'input>>::SLICE,\n    pub data2: <[u8; 2] as flat_serialize::FlatSerializable<'input>>::SLICE,\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u32 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n            as usize];\n    current_size += <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <[u16; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8 as usize];\n    if <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n        min_align = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n    }\n    min_align = match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <[u8; 2] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<[u8; 2] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n            as usize];\n    if <[u8; 2] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n        min_align = <[u8; 2] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n    }\n    min_align = match <[u8; 2] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n};\nconst _: () = {\n    fn header<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = header::<u64>;\n    fn data_len<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = data_len::<u32>;\n    fn array<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = array::<[u16; 3]>;\n    fn data<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = data::<u8>;\n    fn data2<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = data2::<[u8; 2]>;\n};\nunsafe impl<'input> flat_serialize::FlatSerializable<'input> for Basic<'input> {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment = 1;\n        let alignment = <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <[u8; 2] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::align_of;\n        let mut min_align: Option<usize> = None;\n        let ty_align = <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = <[u16; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = { Some(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT) };\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = { Some(<[u8; 2] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT) };\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        match min_align {\n            None => None,\n            Some(min_align) => {\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            }\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size = 0;\n        size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <u32 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += 0;\n        size += 0;\n        size\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'input, Basic<'input>>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(\n        mut input: &'input [u8],\n    ) -> Result<(Self, &'input [u8]), flat_serialize::WrapErr> {\n        if input.len() < Self::MIN_LEN {\n            return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN));\n        }\n        let __packet_macro_read_len = 0usize;\n        let mut header: Option<u64> = None;\n        let mut data_len: Option<u32> = None;\n        let mut array: Option<[u16; 3]> = None;\n        let mut data: Option<<u8 as flat_serialize::FlatSerializable<'_>>::SLICE> = None;\n        let mut data2: Option<<[u8; 2] as flat_serialize::FlatSerializable<'_>>::SLICE> = None;\n        'tryref: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                header = Some(field);\n            }\n            {\n                let (field, rem) = match <u32>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                data_len = Some(field);\n            }\n            {\n                let (field, rem) = match <[u16; 3]>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                array = Some(field);\n            }\n            {\n                let count = (data_len.clone().unwrap()) as usize;\n                let (field, rem) = match <_ as flat_serialize::Slice<'_>>::try_ref(input, count) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                data = Some(field);\n            }\n            {\n                let count = (data_len.clone().unwrap() / 3) as usize;\n                let (field, rem) = match <_ as flat_serialize::Slice<'_>>::try_ref(input, count) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                data2 = Some(field);\n            }\n            let _ref = Basic {\n                header: header.unwrap(),\n                data_len: data_len.unwrap(),\n                array: array.unwrap(),\n                data: data.unwrap(),\n                data2: data2.unwrap(),\n            };\n            return Ok((_ref, input));\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            0 + <u64>::MIN_LEN\n                + <u32>::MIN_LEN\n                + <[u16; 3]>::MIN_LEN\n                + (|| {\n                    <u8>::MIN_LEN\n                        * (match data_len {\n                            Some(data_len) => data_len,\n                            None => return 0usize,\n                        }) as usize\n                })()\n                + (|| {\n                    <[u8; 2]>::MIN_LEN\n                        * (match data_len {\n                            Some(data_len) => data_len,\n                            None => return 0usize,\n                        } / 3) as usize\n                })(),\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        let Basic {\n            header,\n            data_len,\n            array,\n            data,\n            data2,\n        } = self;\n        unsafe {\n            input = header.fill_slice(input);\n        };\n        unsafe {\n            input = data_len.fill_slice(input);\n        };\n        unsafe {\n            input = array.fill_slice(input);\n        };\n        unsafe {\n            let count = (*data_len) as usize;\n            input = <_ as flat_serialize::Slice<'_>>::fill_slice(data, count, input);\n        };\n        unsafe {\n            let count = ((*data_len) / 3) as usize;\n            input = <_ as flat_serialize::Slice<'_>>::fill_slice(data2, count, input);\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    fn len(&self) -> usize {\n        let Basic {\n            header,\n            data_len,\n            array,\n            data,\n            data2,\n        } = self;\n        0usize\n            + <u64 as flat_serialize::FlatSerializable>::len(header)\n            + <u32 as flat_serialize::FlatSerializable>::len(data_len)\n            + <[u16; 3] as flat_serialize::FlatSerializable>::len(array)\n            + (<_ as flat_serialize::Slice<'_>>::len(data, (*data_len) as usize))\n            + (<_ as flat_serialize::Slice<'_>>::len(data2, ((*data_len) / 3) as usize))\n    }\n}\n#[derive(Clone, Debug, PartialEq, Eq)]\npub struct Optional {\n    pub header: u64,\n    pub optional_field: Option<u32>,\n    pub non_optional_field: u16,\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    if <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n        min_align = <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n    }\n    min_align = match <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <u16 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u16 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u16 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u16 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n};\nconst _: () = {\n    fn header<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = header::<u64>;\n    fn optional_field<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = optional_field::<u32>;\n    fn non_optional_field<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = non_optional_field::<u16>;\n};\nunsafe impl<'a> flat_serialize::FlatSerializable<'a> for Optional {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment = 1;\n        let alignment = <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <u16 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::align_of;\n        let mut min_align: Option<usize> = None;\n        let ty_align = <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = {\n            let ty_provied = <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match ty_provied {\n                Some(align) => Some(align),\n                None => Some(<u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT),\n            }\n        };\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = <u16 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        match min_align {\n            None => None,\n            Some(min_align) => {\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            }\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size = 0;\n        size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += 0;\n        size += <u16 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'a, Optional>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(mut input: &[u8]) -> Result<(Self, &[u8]), flat_serialize::WrapErr> {\n        if input.len() < Self::MIN_LEN {\n            return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN));\n        }\n        let __packet_macro_read_len = 0usize;\n        let mut header: Option<u64> = None;\n        let mut optional_field: Option<u32> = None;\n        let mut non_optional_field: Option<u16> = None;\n        'tryref: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                header = Some(field);\n            }\n            if header.clone().unwrap() != 1 {\n                let (field, rem) = match <u32>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                optional_field = Some(field);\n            }\n            {\n                let (field, rem) = match <u16>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                non_optional_field = Some(field);\n            }\n            let _ref = Optional {\n                header: header.unwrap(),\n                optional_field: optional_field,\n                non_optional_field: non_optional_field.unwrap(),\n            };\n            return Ok((_ref, input));\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            0 + <u64>::MIN_LEN\n                + (|| {\n                    if match header {\n                        Some(header) => header,\n                        None => return 0usize,\n                    } != 1\n                    {\n                        <u32>::MIN_LEN\n                    } else {\n                        0\n                    }\n                })()\n                + <u16>::MIN_LEN,\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        let Optional {\n            header,\n            optional_field,\n            non_optional_field,\n        } = self;\n        unsafe {\n            input = header.fill_slice(input);\n        };\n        unsafe {\n            if (*header) != 1 {\n                let optional_field: &u32 = optional_field.as_ref().unwrap();\n                input = optional_field.fill_slice(input);\n            }\n        };\n        unsafe {\n            input = non_optional_field.fill_slice(input);\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    fn len(&self) -> usize {\n        let Optional {\n            header,\n            optional_field,\n            non_optional_field,\n        } = self;\n        0usize\n            + <u64 as flat_serialize::FlatSerializable>::len(header)\n            + (if (*header) != 1 {\n                <u32 as flat_serialize::FlatSerializable>::len(optional_field.as_ref().unwrap())\n            } else {\n                0\n            })\n            + <u16 as flat_serialize::FlatSerializable>::len(non_optional_field)\n    }\n}\n#[derive(Clone, Debug)]\npub struct Nested<'a> {\n    pub prefix: u64,\n    pub basic: Basic<'a>,\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <Basic as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<Basic as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n            as usize];\n    current_size += <Basic as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <Basic as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n};\nconst _: () = {\n    fn prefix<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = prefix::<u64>;\n    fn basic<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = basic::<Basic<'static>>;\n};\nunsafe impl<'a> flat_serialize::FlatSerializable<'a> for Nested<'a> {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment = 1;\n        let alignment = <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <Basic as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::align_of;\n        let mut min_align: Option<usize> = None;\n        let ty_align = <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = <Basic as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        match min_align {\n            None => None,\n            Some(min_align) => {\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            }\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size = 0;\n        size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <Basic as flat_serialize::FlatSerializable>::MIN_LEN;\n        size\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'a, Nested<'a>>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(mut input: &'a [u8]) -> Result<(Self, &'a [u8]), flat_serialize::WrapErr> {\n        if input.len() < Self::MIN_LEN {\n            return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN));\n        }\n        let __packet_macro_read_len = 0usize;\n        let mut prefix: Option<u64> = None;\n        let mut basic: Option<Basic<'a>> = None;\n        'tryref: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                prefix = Some(field);\n            }\n            {\n                let (field, rem) = match <Basic>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                basic = Some(field);\n            }\n            let _ref = Nested {\n                prefix: prefix.unwrap(),\n                basic: basic.unwrap(),\n            };\n            return Ok((_ref, input));\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            0 + <u64>::MIN_LEN + <Basic>::MIN_LEN,\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        let Nested { prefix, basic } = self;\n        unsafe {\n            input = prefix.fill_slice(input);\n        };\n        unsafe {\n            input = basic.fill_slice(input);\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    fn len(&self) -> usize {\n        let Nested { prefix, basic } = self;\n        0usize\n            + <u64 as flat_serialize::FlatSerializable>::len(prefix)\n            + <Basic as flat_serialize::FlatSerializable>::len(basic)\n    }\n}\n#[derive(Clone, Debug)]\npub struct NestedOptional {\n    pub present: u64,\n    pub val: Option<Optional>,\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n            as usize];\n    if <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n        min_align = <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n    }\n    min_align = match <Optional as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n};\nconst _: () = {\n    fn present<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = present::<u64>;\n    fn val<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = val::<Optional>;\n};\nunsafe impl<'a> flat_serialize::FlatSerializable<'a> for NestedOptional {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment = 1;\n        let alignment = <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::align_of;\n        let mut min_align: Option<usize> = None;\n        let ty_align = <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = {\n            let ty_provied = <Optional as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match ty_provied {\n                Some(align) => Some(align),\n                None => Some(<Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT),\n            }\n        };\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        match min_align {\n            None => None,\n            Some(min_align) => {\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            }\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size = 0;\n        size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += 0;\n        size\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'a, NestedOptional>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(mut input: &[u8]) -> Result<(Self, &[u8]), flat_serialize::WrapErr> {\n        if input.len() < Self::MIN_LEN {\n            return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN));\n        }\n        let __packet_macro_read_len = 0usize;\n        let mut present: Option<u64> = None;\n        let mut val: Option<Optional> = None;\n        'tryref: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                present = Some(field);\n            }\n            if present.clone().unwrap() > 2 {\n                let (field, rem) = match <Optional>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                val = Some(field);\n            }\n            let _ref = NestedOptional {\n                present: present.unwrap(),\n                val: val,\n            };\n            return Ok((_ref, input));\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            0 + <u64>::MIN_LEN\n                + (|| {\n                    if match present {\n                        Some(present) => present,\n                        None => return 0usize,\n                    } > 2\n                    {\n                        <Optional>::MIN_LEN\n                    } else {\n                        0\n                    }\n                })(),\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        let NestedOptional { present, val } = self;\n        unsafe {\n            input = present.fill_slice(input);\n        };\n        unsafe {\n            if (*present) > 2 {\n                let val: &Optional = val.as_ref().unwrap();\n                input = val.fill_slice(input);\n            }\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    fn len(&self) -> usize {\n        let NestedOptional { present, val } = self;\n        0usize\n            + <u64 as flat_serialize::FlatSerializable>::len(present)\n            + (if (*present) > 2 {\n                <Optional as flat_serialize::FlatSerializable>::len(val.as_ref().unwrap())\n            } else {\n                0\n            })\n    }\n}\n#[derive(Clone, Debug)]\npub struct NestedSlice<'b> {\n    pub num_vals: u64,\n    pub vals: <Optional as flat_serialize::FlatSerializable<'b>>::SLICE,\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    let _alignment_check: () =\n        [()][(current_size) % <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n            as usize];\n    if <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n        min_align = <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n    }\n    min_align = match <Optional as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n};\nconst _: () = {\n    fn num_vals<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = num_vals::<u64>;\n    fn vals<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = vals::<Optional>;\n};\nunsafe impl<'b> flat_serialize::FlatSerializable<'b> for NestedSlice<'b> {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment = 1;\n        let alignment = <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::align_of;\n        let mut min_align: Option<usize> = None;\n        let ty_align = <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        let ty_align = { Some(<Optional as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT) };\n        match (ty_align, min_align) {\n            (None, _) => (),\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => (),\n        }\n        match min_align {\n            None => None,\n            Some(min_align) => {\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            }\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size = 0;\n        size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += 0;\n        size\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'b, NestedSlice<'b>>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(mut input: &'b [u8]) -> Result<(Self, &'b [u8]), flat_serialize::WrapErr> {\n        if input.len() < Self::MIN_LEN {\n            return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN));\n        }\n        let __packet_macro_read_len = 0usize;\n        let mut num_vals: Option<u64> = None;\n        let mut vals: Option<<Optional as flat_serialize::FlatSerializable<'_>>::SLICE> = None;\n        'tryref: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                num_vals = Some(field);\n            }\n            {\n                let count = (num_vals.clone().unwrap()) as usize;\n                let (field, rem) = match <_ as flat_serialize::Slice<'_>>::try_ref(input, count) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                vals = Some(field);\n            }\n            let _ref = NestedSlice {\n                num_vals: num_vals.unwrap(),\n                vals: vals.unwrap(),\n            };\n            return Ok((_ref, input));\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            0 + <u64>::MIN_LEN\n                + (|| {\n                    <Optional>::MIN_LEN\n                        * (match num_vals {\n                            Some(num_vals) => num_vals,\n                            None => return 0usize,\n                        }) as usize\n                })(),\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        let NestedSlice { num_vals, vals } = self;\n        unsafe {\n            input = num_vals.fill_slice(input);\n        };\n        unsafe {\n            let count = (*num_vals) as usize;\n            input = <_ as flat_serialize::Slice<'_>>::fill_slice(vals, count, input);\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    fn len(&self) -> usize {\n        let NestedSlice { num_vals, vals } = self;\n        0usize\n            + <u64 as flat_serialize::FlatSerializable>::len(num_vals)\n            + (<_ as flat_serialize::Slice<'_>>::len(vals, (*num_vals) as usize))\n    }\n}\n#[derive(Clone, Debug)]\npub enum BasicEnum<'input> {\n    First {\n        data_len: u32,\n        data: <u8 as flat_serialize::FlatSerializable<'input>>::SLICE,\n    },\n    Fixed {\n        array: [u16; 3],\n    },\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()][(<u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        > min_align) as u8 as usize];\n    current_size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    {\n        use std::mem::{align_of, size_of};\n        let mut current_size = current_size;\n        let mut min_align = min_align;\n        let _alignment_check: () =\n            [()][(current_size) % <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        current_size += <u32 as flat_serialize::FlatSerializable>::MIN_LEN;\n        min_align = match <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n        let _alignment_check: () =\n            [()][(current_size) % <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        if <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n            min_align = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        }\n        min_align = match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n    }\n    {\n        use std::mem::{align_of, size_of};\n        let mut current_size = current_size;\n        let mut min_align = min_align;\n        let _alignment_check: () = [()]\n            [(current_size) % <[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        current_size += <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n        min_align = match <[u16; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n    }\n};\nconst _: () = {\n    #[allow(dead_code)]\n    enum UniquenessCheck {\n        First = 2,\n        Fixed = 3,\n    }\n};\nconst _: () = {\n    fn k<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = k::<u64>;\n    const _: () = {\n        const _: () = {\n            fn data_len<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = data_len::<u32>;\n            fn data<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = data::<u8>;\n        };\n    };\n    const _: () = {\n        const _: () = {\n            fn array<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = array::<[u16; 3]>;\n        };\n    };\n};\nunsafe impl<'input> flat_serialize::FlatSerializable<'input> for BasicEnum<'input> {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment: usize =\n            <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        let alignment: usize = {\n            let mut required_alignment =\n                <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            let alignment = <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            let alignment = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            required_alignment\n        };\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment: usize = {\n            let mut required_alignment =\n                <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            let alignment = <[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            required_alignment\n        };\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::{align_of, size_of};\n        let mut min_align: usize =\n            match match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                Some(a) => Some(a),\n                None => Some(8),\n            } {\n                None => 8,\n                Some(align) => align,\n            };\n        let variant_alignment: usize = {\n            let mut min_align: Option<usize> =\n                match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                    Some(a) => Some(a),\n                    None => Some(8),\n                };\n            let alignment = <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let alignment = { Some(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT) };\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let variant_size: usize = <u64 as flat_serialize::FlatSerializable>::MIN_LEN\n                + <u32 as flat_serialize::FlatSerializable>::MIN_LEN\n                + 0;\n            let effective_alignment = match min_align {\n                Some(align) => align,\n                None => 8,\n            };\n            if variant_size % 8 == 0 && effective_alignment >= 8 {\n                8\n            } else if variant_size % 4 == 0 && effective_alignment >= 4 {\n                4\n            } else if variant_size % 2 == 0 && effective_alignment >= 2 {\n                2\n            } else {\n                1\n            }\n        };\n        if variant_alignment < min_align {\n            min_align = variant_alignment\n        }\n        let variant_alignment: usize = {\n            let mut min_align: Option<usize> =\n                match <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                    Some(a) => Some(a),\n                    None => Some(8),\n                };\n            let alignment = <[u16; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let variant_size: usize = <u64 as flat_serialize::FlatSerializable>::MIN_LEN\n                + <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n            let effective_alignment = match min_align {\n                Some(align) => align,\n                None => 8,\n            };\n            if variant_size % 8 == 0 && effective_alignment >= 8 {\n                8\n            } else if variant_size % 4 == 0 && effective_alignment >= 4 {\n                4\n            } else if variant_size % 2 == 0 && effective_alignment >= 2 {\n                2\n            } else {\n                1\n            }\n        };\n        if variant_alignment < min_align {\n            min_align = variant_alignment\n        }\n        let min_size = Self::MIN_LEN;\n        if min_size % 8 == 0 && min_align >= 8 {\n            Some(8)\n        } else if min_size % 4 == 0 && min_align >= 4 {\n            Some(4)\n        } else if min_size % 2 == 0 && min_align >= 2 {\n            Some(2)\n        } else {\n            Some(1)\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size: Option<usize> = None;\n        let variant_size = {\n            let mut size: usize = <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += <u32 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += 0;\n            size\n        };\n        size = match size {\n            None => Some(variant_size),\n            Some(size) if size > variant_size => Some(variant_size),\n            Some(size) => Some(size),\n        };\n        let variant_size = {\n            let mut size: usize = <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n            size\n        };\n        size = match size {\n            None => Some(variant_size),\n            Some(size) if size > variant_size => Some(variant_size),\n            Some(size) => Some(size),\n        };\n        match size {\n            Some(size) => size,\n            None => <u64 as flat_serialize::FlatSerializable>::MIN_LEN,\n        }\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'input, BasicEnum<'input>>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(\n        mut input: &'input [u8],\n    ) -> Result<(Self, &'input [u8]), flat_serialize::WrapErr> {\n        let __packet_macro_read_len = 0usize;\n        let mut k = None;\n        'tryref_tag: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref_tag,\n                };\n                input = rem;\n                k = Some(field);\n            };\n            match k {\n                Some(2) => {\n                    let mut data_len: Option<u32> = None;\n                    let mut data: Option<<u8 as flat_serialize::FlatSerializable<'_>>::SLICE> =\n                        None;\n                    'tryref_0: loop {\n                        {\n                            let (field, rem) = match <u32>::try_ref(input) {\n                                Ok((f, b)) => (f, b),\n                                Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                    return Err(flat_serialize::WrapErr::InvalidTag(\n                                        __packet_macro_read_len + offset,\n                                    ))\n                                }\n                                Err(..) => break 'tryref_0,\n                            };\n                            input = rem;\n                            data_len = Some(field);\n                        }\n                        {\n                            let count = (data_len.clone().unwrap()) as usize;\n                            let (field, rem) =\n                                match <_ as flat_serialize::Slice<'_>>::try_ref(input, count) {\n                                    Ok((f, b)) => (f, b),\n                                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                        return Err(flat_serialize::WrapErr::InvalidTag(\n                                            __packet_macro_read_len + offset,\n                                        ))\n                                    }\n                                    Err(..) => break 'tryref_0,\n                                };\n                            input = rem;\n                            data = Some(field);\n                        }\n                        let _ref = BasicEnum::First {\n                            data_len: data_len.unwrap(),\n                            data: data.unwrap(),\n                        };\n                        return Ok((_ref, input));\n                    }\n                    return Err(flat_serialize::WrapErr::NotEnoughBytes(\n                        std::mem::size_of::<u64>()\n                            + <u32>::MIN_LEN\n                            + (|| {\n                                <u8>::MIN_LEN\n                                    * (match data_len {\n                                        Some(data_len) => data_len,\n                                        None => return 0usize,\n                                    }) as usize\n                            })(),\n                    ));\n                }\n                Some(3) => {\n                    let mut array: Option<[u16; 3]> = None;\n                    'tryref_1: loop {\n                        {\n                            let (field, rem) = match <[u16; 3]>::try_ref(input) {\n                                Ok((f, b)) => (f, b),\n                                Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                    return Err(flat_serialize::WrapErr::InvalidTag(\n                                        __packet_macro_read_len + offset,\n                                    ))\n                                }\n                                Err(..) => break 'tryref_1,\n                            };\n                            input = rem;\n                            array = Some(field);\n                        }\n                        let _ref = BasicEnum::Fixed {\n                            array: array.unwrap(),\n                        };\n                        return Ok((_ref, input));\n                    }\n                    return Err(flat_serialize::WrapErr::NotEnoughBytes(\n                        std::mem::size_of::<u64>() + <[u16; 3]>::MIN_LEN,\n                    ));\n                }\n                _ => return Err(flat_serialize::WrapErr::InvalidTag(0)),\n            }\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            ::std::mem::size_of::<u64>(),\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        match self {\n            BasicEnum::First { data_len, data } => {\n                let k: &u64 = &2;\n                unsafe {\n                    input = k.fill_slice(input);\n                }\n                unsafe {\n                    input = data_len.fill_slice(input);\n                };\n                unsafe {\n                    let count = (*data_len) as usize;\n                    input = <_ as flat_serialize::Slice<'_>>::fill_slice(data, count, input);\n                }\n            }\n            BasicEnum::Fixed { array } => {\n                let k: &u64 = &3;\n                unsafe {\n                    input = k.fill_slice(input);\n                }\n                unsafe {\n                    input = array.fill_slice(input);\n                }\n            }\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    fn len(&self) -> usize {\n        match self {\n            BasicEnum::First { data_len, data } => {\n                ::std::mem::size_of::<u64>()\n                    + <u32 as flat_serialize::FlatSerializable>::len(data_len)\n                    + (<_ as flat_serialize::Slice<'_>>::len(data, (*data_len) as usize))\n            }\n            BasicEnum::Fixed { array } => {\n                ::std::mem::size_of::<u64>()\n                    + <[u16; 3] as flat_serialize::FlatSerializable>::len(array)\n            }\n        }\n    }\n}\n#[derive(Clone, Debug)]\npub enum PaddedEnum<'input> {\n    First {\n        padding: [u8; 3],\n        data_len: u32,\n        data: <u8 as flat_serialize::FlatSerializable<'input>>::SLICE,\n    },\n    Fixed {\n        padding: u8,\n        array: [u16; 3],\n    },\n}\n#[allow(unused_assignments)]\nconst _: () = {\n    use std::mem::{align_of, size_of};\n    let mut current_size = 0;\n    let mut min_align = 8;\n    let _alignment_check: () =\n        [()][(current_size) % <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n    let _alignment_check2: () = [()]\n        [(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8 as usize];\n    current_size += <u8 as flat_serialize::FlatSerializable>::MIN_LEN;\n    min_align = match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n        Some(align) if align < min_align => align,\n        _ => min_align,\n    };\n    {\n        use std::mem::{align_of, size_of};\n        let mut current_size = current_size;\n        let mut min_align = min_align;\n        let _alignment_check: () = [()]\n            [(current_size) % <[u8; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<[u8; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        current_size += <[u8; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n        min_align = match <[u8; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n        let _alignment_check: () =\n            [()][(current_size) % <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        current_size += <u32 as flat_serialize::FlatSerializable>::MIN_LEN;\n        min_align = match <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n        let _alignment_check: () =\n            [()][(current_size) % <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        if <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < min_align {\n            min_align = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        }\n        min_align = match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n    }\n    {\n        use std::mem::{align_of, size_of};\n        let mut current_size = current_size;\n        let mut min_align = min_align;\n        let _alignment_check: () =\n            [()][(current_size) % <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        current_size += <u8 as flat_serialize::FlatSerializable>::MIN_LEN;\n        min_align = match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n        let _alignment_check: () = [()]\n            [(current_size) % <[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT];\n        let _alignment_check2: () = [()]\n            [(<[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > min_align) as u8\n                as usize];\n        current_size += <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n        min_align = match <[u16; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n            Some(align) if align < min_align => align,\n            _ => min_align,\n        };\n    }\n};\nconst _: () = {\n    #[allow(dead_code)]\n    enum UniquenessCheck {\n        First = 2,\n        Fixed = 3,\n    }\n};\nconst _: () = {\n    fn k<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n    let _ = k::<u8>;\n    const _: () = {\n        const _: () = {\n            fn padding<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = padding::<[u8; 3]>;\n            fn data_len<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = data_len::<u32>;\n            fn data<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = data::<u8>;\n        };\n    };\n    const _: () = {\n        const _: () = {\n            fn padding<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = padding::<u8>;\n            fn array<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = array::<[u16; 3]>;\n        };\n    };\n};\nunsafe impl<'input> flat_serialize::FlatSerializable<'input> for PaddedEnum<'input> {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment: usize =\n            <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        let alignment: usize = {\n            let mut required_alignment =\n                <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            let alignment = <[u8; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            let alignment = <u32 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            let alignment = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            required_alignment\n        };\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment: usize = {\n            let mut required_alignment =\n                <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            let alignment = <u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            let alignment = <[u16; 3] as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n            if alignment > required_alignment {\n                required_alignment = alignment;\n            }\n            required_alignment\n        };\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::{align_of, size_of};\n        let mut min_align: usize =\n            match match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                Some(a) => Some(a),\n                None => Some(8),\n            } {\n                None => 8,\n                Some(align) => align,\n            };\n        let variant_alignment: usize = {\n            let mut min_align: Option<usize> =\n                match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                    Some(a) => Some(a),\n                    None => Some(8),\n                };\n            let alignment = <[u8; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let alignment = <u32 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let alignment = { Some(<u8 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT) };\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let variant_size: usize = <u8 as flat_serialize::FlatSerializable>::MIN_LEN\n                + <[u8; 3] as flat_serialize::FlatSerializable>::MIN_LEN\n                + <u32 as flat_serialize::FlatSerializable>::MIN_LEN\n                + 0;\n            let effective_alignment = match min_align {\n                Some(align) => align,\n                None => 8,\n            };\n            if variant_size % 8 == 0 && effective_alignment >= 8 {\n                8\n            } else if variant_size % 4 == 0 && effective_alignment >= 4 {\n                4\n            } else if variant_size % 2 == 0 && effective_alignment >= 2 {\n                2\n            } else {\n                1\n            }\n        };\n        if variant_alignment < min_align {\n            min_align = variant_alignment\n        }\n        let variant_alignment: usize = {\n            let mut min_align: Option<usize> =\n                match <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                    Some(a) => Some(a),\n                    None => Some(8),\n                };\n            let alignment = <u8 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let alignment = <[u16; 3] as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n            match (alignment, min_align) {\n                (None, _) => (),\n                (Some(align), None) => min_align = Some(align),\n                (Some(align), Some(min)) if align < min => min_align = Some(align),\n                _ => (),\n            }\n            let variant_size: usize = <u8 as flat_serialize::FlatSerializable>::MIN_LEN\n                + <u8 as flat_serialize::FlatSerializable>::MIN_LEN\n                + <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n            let effective_alignment = match min_align {\n                Some(align) => align,\n                None => 8,\n            };\n            if variant_size % 8 == 0 && effective_alignment >= 8 {\n                8\n            } else if variant_size % 4 == 0 && effective_alignment >= 4 {\n                4\n            } else if variant_size % 2 == 0 && effective_alignment >= 2 {\n                2\n            } else {\n                1\n            }\n        };\n        if variant_alignment < min_align {\n            min_align = variant_alignment\n        }\n        let min_size = Self::MIN_LEN;\n        if min_size % 8 == 0 && min_align >= 8 {\n            Some(8)\n        } else if min_size % 4 == 0 && min_align >= 4 {\n            Some(4)\n        } else if min_size % 2 == 0 && min_align >= 2 {\n            Some(2)\n        } else {\n            Some(1)\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size: Option<usize> = None;\n        let variant_size = {\n            let mut size: usize = <u8 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += <[u8; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += <u32 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += 0;\n            size\n        };\n        size = match size {\n            None => Some(variant_size),\n            Some(size) if size > variant_size => Some(variant_size),\n            Some(size) => Some(size),\n        };\n        let variant_size = {\n            let mut size: usize = <u8 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += <u8 as flat_serialize::FlatSerializable>::MIN_LEN;\n            size += <[u16; 3] as flat_serialize::FlatSerializable>::MIN_LEN;\n            size\n        };\n        size = match size {\n            None => Some(variant_size),\n            Some(size) if size > variant_size => Some(variant_size),\n            Some(size) => Some(size),\n        };\n        match size {\n            Some(size) => size,\n            None => <u8 as flat_serialize::FlatSerializable>::MIN_LEN,\n        }\n    };\n    const TRIVIAL_COPY: bool = false;\n    type SLICE = flat_serialize::Iterable<'input, PaddedEnum<'input>>;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(\n        mut input: &'input [u8],\n    ) -> Result<(Self, &'input [u8]), flat_serialize::WrapErr> {\n        let __packet_macro_read_len = 0usize;\n        let mut k = None;\n        'tryref_tag: loop {\n            {\n                let (field, rem) = match <u8>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ))\n                    }\n                    Err(..) => break 'tryref_tag,\n                };\n                input = rem;\n                k = Some(field);\n            };\n            match k {\n                Some(2) => {\n                    let mut padding: Option<[u8; 3]> = None;\n                    let mut data_len: Option<u32> = None;\n                    let mut data: Option<<u8 as flat_serialize::FlatSerializable<'_>>::SLICE> =\n                        None;\n                    'tryref_0: loop {\n                        {\n                            let (field, rem) = match <[u8; 3]>::try_ref(input) {\n                                Ok((f, b)) => (f, b),\n                                Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                    return Err(flat_serialize::WrapErr::InvalidTag(\n                                        __packet_macro_read_len + offset,\n                                    ))\n                                }\n                                Err(..) => break 'tryref_0,\n                            };\n                            input = rem;\n                            padding = Some(field);\n                        }\n                        {\n                            let (field, rem) = match <u32>::try_ref(input) {\n                                Ok((f, b)) => (f, b),\n                                Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                    return Err(flat_serialize::WrapErr::InvalidTag(\n                                        __packet_macro_read_len + offset,\n                                    ))\n                                }\n                                Err(..) => break 'tryref_0,\n                            };\n                            input = rem;\n                            data_len = Some(field);\n                        }\n                        {\n                            let count = (data_len.clone().unwrap()) as usize;\n                            let (field, rem) =\n                                match <_ as flat_serialize::Slice<'_>>::try_ref(input, count) {\n                                    Ok((f, b)) => (f, b),\n                                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                        return Err(flat_serialize::WrapErr::InvalidTag(\n                                            __packet_macro_read_len + offset,\n                                        ))\n                                    }\n                                    Err(..) => break 'tryref_0,\n                                };\n                            input = rem;\n                            data = Some(field);\n                        }\n                        let _ref = PaddedEnum::First {\n                            padding: padding.unwrap(),\n                            data_len: data_len.unwrap(),\n                            data: data.unwrap(),\n                        };\n                        return Ok((_ref, input));\n                    }\n                    return Err(flat_serialize::WrapErr::NotEnoughBytes(\n                        std::mem::size_of::<u8>()\n                            + <[u8; 3]>::MIN_LEN\n                            + <u32>::MIN_LEN\n                            + (|| {\n                                <u8>::MIN_LEN\n                                    * (match data_len {\n                                        Some(data_len) => data_len,\n                                        None => return 0usize,\n                                    }) as usize\n                            })(),\n                    ));\n                }\n                Some(3) => {\n                    let mut padding: Option<u8> = None;\n                    let mut array: Option<[u16; 3]> = None;\n                    'tryref_1: loop {\n                        {\n                            let (field, rem) = match <u8>::try_ref(input) {\n                                Ok((f, b)) => (f, b),\n                                Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                    return Err(flat_serialize::WrapErr::InvalidTag(\n                                        __packet_macro_read_len + offset,\n                                    ))\n                                }\n                                Err(..) => break 'tryref_1,\n                            };\n                            input = rem;\n                            padding = Some(field);\n                        }\n                        {\n                            let (field, rem) = match <[u16; 3]>::try_ref(input) {\n                                Ok((f, b)) => (f, b),\n                                Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                                    return Err(flat_serialize::WrapErr::InvalidTag(\n                                        __packet_macro_read_len + offset,\n                                    ))\n                                }\n                                Err(..) => break 'tryref_1,\n                            };\n                            input = rem;\n                            array = Some(field);\n                        }\n                        let _ref = PaddedEnum::Fixed {\n                            padding: padding.unwrap(),\n                            array: array.unwrap(),\n                        };\n                        return Ok((_ref, input));\n                    }\n                    return Err(flat_serialize::WrapErr::NotEnoughBytes(\n                        std::mem::size_of::<u8>() + <u8>::MIN_LEN + <[u16; 3]>::MIN_LEN,\n                    ));\n                }\n                _ => return Err(flat_serialize::WrapErr::InvalidTag(0)),\n            }\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            ::std::mem::size_of::<u8>(),\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.len();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        match self {\n            PaddedEnum::First {\n                padding,\n                data_len,\n                data,\n            } => {\n                let k: &u8 = &2;\n                unsafe {\n                    input = k.fill_slice(input);\n                }\n                unsafe {\n                    input = padding.fill_slice(input);\n                };\n                unsafe {\n                    input = data_len.fill_slice(input);\n                };\n                unsafe {\n                    let count = (*data_len) as usize;\n                    input = <_ as flat_serialize::Slice<'_>>::fill_slice(data, count, input);\n                }\n            }\n            PaddedEnum::Fixed { padding, array } => {\n                let k: &u8 = &3;\n                unsafe {\n                    input = k.fill_slice(input);\n                }\n                unsafe {\n                    input = padding.fill_slice(input);\n                };\n                unsafe {\n                    input = array.fill_slice(input);\n                }\n            }\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    fn len(&self) -> usize {\n        match self {\n            PaddedEnum::First {\n                padding,\n                data_len,\n                data,\n            } => {\n                ::std::mem::size_of::<u8>()\n                    + <[u8; 3] as flat_serialize::FlatSerializable>::len(padding)\n                    + <u32 as flat_serialize::FlatSerializable>::len(data_len)\n                    + (<_ as flat_serialize::Slice<'_>>::len(data, (*data_len) as usize))\n            }\n            PaddedEnum::Fixed { padding, array } => {\n                ::std::mem::size_of::<u8>()\n                    + <u8 as flat_serialize::FlatSerializable>::len(padding)\n                    + <[u16; 3] as flat_serialize::FlatSerializable>::len(array)\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "crates/flat_serialize/flat_serialize/Cargo.toml",
    "content": "[package]\nname = \"flat_serialize\"\nversion = \"0.1.0\"\nauthors = [\"Joshua Lockerman\"]\nedition = \"2021\"\n\n[dependencies]\nordered-float = \"1.0\"\nserde = \"1.0\"\n\n[dev-dependencies]\nflat_serialize_macro = {path=\"../flat_serialize_macro\"}\n"
  },
  {
    "path": "crates/flat_serialize/flat_serialize/src/lib.rs",
    "content": "use std::{\n    fmt,\n    marker::PhantomData,\n    mem::{align_of, size_of, MaybeUninit},\n    slice,\n};\n#[derive(Debug)]\npub enum WrapErr {\n    NotEnoughBytes(usize),\n    InvalidTag(usize),\n}\n\n/// Trait marking that a type can be translated to and from a flat buffer\n/// without copying or allocation.\n///\n/// # Safety\n/// For a type to be `FlatSerializable` it must contain no pointers, have no\n/// interior padding, must have a `size >= alignment` and must have\n/// `size % align = 0`. In general this should not be implemented manually, and\n/// you should only use `#[derive(FlatSerializable)]` or `flat_serialize!{}` to\n/// implement this.\n/// **NOTE** we currently allow types with invalid bit patterns, such as `bool`\n/// to be `FlatSerializable` making this trait inappropriate to use on untrusted\n/// input.\npub unsafe trait FlatSerializable<'input>: Sized + 'input {\n    const MIN_LEN: usize;\n    const REQUIRED_ALIGNMENT: usize;\n    const MAX_PROVIDED_ALIGNMENT: Option<usize>;\n    const TRIVIAL_COPY: bool = false;\n    type SLICE;\n    type OWNED: 'static;\n\n    #[allow(clippy::missing_safety_doc)]\n    unsafe fn try_ref(input: &'input [u8]) -> Result<(Self, &'input [u8]), WrapErr>;\n    fn fill_vec(&self, input: &mut Vec<u8>) {\n        let start = input.len();\n        let my_len = self.num_bytes();\n        input.reserve(my_len);\n        // simulate unstable spare_capacity_mut()\n        let slice = unsafe {\n            slice::from_raw_parts_mut(\n                input.as_mut_ptr().add(input.len()) as *mut MaybeUninit<u8>,\n                my_len,\n            )\n        };\n        let rem = unsafe { self.fill_slice(slice) };\n        debug_assert_eq!(rem.len(), 0);\n        unsafe {\n            input.set_len(start + my_len);\n        }\n    }\n    #[must_use]\n    #[allow(clippy::missing_safety_doc)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [MaybeUninit<u8>],\n    ) -> &'out mut [MaybeUninit<u8>];\n    fn num_bytes(&self) -> usize;\n\n    fn make_owned(&mut self);\n    fn into_owned(self) -> Self::OWNED;\n}\n\n#[macro_export]\nmacro_rules! impl_flat_serializable {\n    ($($typ:ty)+) => {\n        $(\n            unsafe impl<'i> FlatSerializable<'i> for $typ {\n                const MIN_LEN: usize = size_of::<Self>();\n                const REQUIRED_ALIGNMENT: usize = align_of::<Self>();\n                const MAX_PROVIDED_ALIGNMENT: Option<usize> = None;\n                const TRIVIAL_COPY: bool = true;\n                type SLICE = $crate::Slice<'i, $typ>;\n                type OWNED = Self;\n\n                #[inline(always)]\n                unsafe fn try_ref(input: &'i [u8])\n                -> Result<(Self, &'i [u8]), WrapErr> {\n                    let size = size_of::<Self>();\n                    if input.len() < size {\n                        return Err(WrapErr::NotEnoughBytes(size))\n                    }\n                    let (field, rem) = input.split_at(size);\n                    let field = field.as_ptr().cast::<Self>();\n                    Ok((field.read_unaligned(), rem))\n                }\n\n                #[inline(always)]\n                unsafe fn fill_slice<'out>(&self, input: &'out mut [MaybeUninit<u8>])\n                -> &'out mut [MaybeUninit<u8>] {\n                    let size = size_of::<Self>();\n                    let (input, rem) = input.split_at_mut(size);\n                    let bytes = (self as *const Self).cast::<MaybeUninit<u8>>();\n                    let bytes = slice::from_raw_parts(bytes, size);\n                    // emulate write_slice_cloned()\n                    // for i in 0..size {\n                    //     input[i] = MaybeUninit::new(bytes[i])\n                    // }\n                    input.copy_from_slice(bytes);\n                    rem\n                }\n\n                #[inline(always)]\n                fn num_bytes(&self) -> usize {\n                    size_of::<Self>()\n                }\n\n                #[inline(always)]\n                fn make_owned(&mut self) {\n                    // nop\n                }\n\n                #[inline(always)]\n                fn into_owned(self) -> Self::OWNED {\n                    self\n                }\n            }\n        )+\n    };\n}\n\nimpl_flat_serializable!(bool);\nimpl_flat_serializable!(i8 u8 i16 u16 i32 u32 i64 u64 i128 u128);\nimpl_flat_serializable!(f32 f64 ordered_float::OrderedFloat<f32> ordered_float::OrderedFloat<f64>);\n\n// TODO ensure perf\nunsafe impl<'i, T, const N: usize> FlatSerializable<'i> for [T; N]\nwhere\n    T: FlatSerializable<'i> + 'i,\n{\n    const MIN_LEN: usize = { T::MIN_LEN * N };\n    const REQUIRED_ALIGNMENT: usize = T::REQUIRED_ALIGNMENT;\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = T::MAX_PROVIDED_ALIGNMENT;\n    const TRIVIAL_COPY: bool = T::TRIVIAL_COPY;\n    // FIXME ensure no padding\n    type SLICE = Slice<'i, [T; N]>;\n    type OWNED = [T::OWNED; N];\n\n    #[inline(always)]\n    unsafe fn try_ref(mut input: &'i [u8]) -> Result<(Self, &'i [u8]), WrapErr> {\n        // TODO can we simplify based on T::TRIVIAL_COPY?\n        if T::TRIVIAL_COPY && input.len() < (T::MIN_LEN * N) {\n            return Err(WrapErr::NotEnoughBytes(T::MIN_LEN * N));\n        }\n        let mut output: [MaybeUninit<T>; N] = MaybeUninit::uninit().assume_init();\n        for item in output.iter_mut() {\n            let (val, rem) = T::try_ref(input)?;\n            *item = MaybeUninit::new(val);\n            input = rem;\n        }\n        let output = (&output as *const [MaybeUninit<T>; N])\n            .cast::<[T; N]>()\n            .read();\n        Ok((output, input))\n    }\n\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [MaybeUninit<u8>],\n    ) -> &'out mut [MaybeUninit<u8>] {\n        let size = if Self::TRIVIAL_COPY {\n            Self::MIN_LEN\n        } else {\n            self.len()\n        };\n        let (mut input, rem) = input.split_at_mut(size);\n        input = &mut input[..size];\n        // TODO is there a way to force a memcopy for trivial cases?\n\n        for val in self {\n            input = val.fill_slice(input);\n        }\n        debug_assert_eq!(input.len(), 0);\n        rem\n    }\n\n    #[inline(always)]\n    fn num_bytes(&self) -> usize {\n        self.iter().map(T::num_bytes).sum()\n    }\n\n    fn make_owned(&mut self) {\n        for val in self {\n            val.make_owned()\n        }\n    }\n\n    fn into_owned(self) -> Self::OWNED {\n        let mut output: [MaybeUninit<T::OWNED>; N] = unsafe { MaybeUninit::uninit().assume_init() };\n        for (i, t) in self.into_iter().map(|s| s.into_owned()).enumerate() {\n            output[i] = MaybeUninit::new(t)\n        }\n\n        unsafe {\n            (&mut output as *mut [MaybeUninit<T::OWNED>; N])\n                .cast::<Self::OWNED>()\n                .read()\n        }\n    }\n}\n\npub enum Slice<'input, T: 'input> {\n    Iter(Unflatten<'input, T>),\n    Slice(&'input [T]),\n    Owned(Vec<T>),\n}\n\nimpl<'input, T: 'input> Slice<'input, T> {\n    pub fn iter<'s>(&'s self) -> Iter<'input, 's, T>\n    where\n        'input: 's,\n    {\n        match self {\n            Slice::Iter(iter) => Iter::Unflatten(*iter),\n            Slice::Slice(slice) => Iter::Slice(slice),\n            Slice::Owned(vec) => Iter::Slice(vec),\n        }\n    }\n}\n\nimpl<'input, T> std::iter::IntoIterator for Slice<'input, T>\nwhere\n    T: FlatSerializable<'input> + Clone,\n{\n    type Item = T;\n\n    type IntoIter = Iter<'input, 'input, T>;\n\n    fn into_iter(self) -> Self::IntoIter {\n        match self {\n            Slice::Iter(iter) => Iter::Unflatten(iter),\n            Slice::Slice(slice) => Iter::Slice(slice),\n            Slice::Owned(vec) => Iter::Owned(vec.into_iter()),\n        }\n    }\n}\n\npub enum Iter<'input, 'borrow, T: 'input> {\n    Unflatten(Unflatten<'input, T>),\n    Slice(&'borrow [T]),\n    Owned(std::vec::IntoIter<T>),\n}\n\nimpl<'input, T: 'input> Iterator for Iter<'input, '_, T>\nwhere\n    T: FlatSerializable<'input> + Clone,\n{\n    type Item = T;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        match self {\n            Self::Unflatten(i) => {\n                if i.slice.is_empty() {\n                    return None;\n                }\n                let (val, rem) = unsafe { <T>::try_ref(i.slice).unwrap() };\n                let additional_len = aligning_len(rem.as_ptr() as _, T::REQUIRED_ALIGNMENT);\n\n                i.slice = &rem[additional_len..];\n                Some(val)\n            }\n            Self::Slice(s) => {\n                let val = s.first().cloned();\n                if val.is_some() {\n                    *s = &s[1..]\n                }\n                val\n            }\n            Self::Owned(i) => i.next(),\n        }\n    }\n\n    fn nth(&mut self, n: usize) -> Option<Self::Item> {\n        match self {\n            Self::Unflatten(i) => {\n                for _ in 0..n {\n                    i.next()?;\n                }\n                i.next()\n            }\n            Self::Slice(s) => {\n                *s = s.get(n..)?;\n                self.next()\n            }\n            Self::Owned(i) => i.nth(n),\n        }\n    }\n}\n\nimpl<'input, T: 'input> Iter<'input, '_, T>\nwhere\n    T: FlatSerializable<'input> + Clone,\n{\n    pub fn len(&self) -> usize {\n        match self {\n            Self::Unflatten(i) => (*i).count(),\n            Self::Slice(s) => s.len(),\n            Self::Owned(i) => i.as_slice().len(),\n        }\n    }\n\n    pub fn is_empty(&self) -> bool {\n        match self {\n            Self::Unflatten(i) => (*i).count() == 0,\n            Self::Slice(s) => s.is_empty(),\n            Self::Owned(i) => i.as_slice().is_empty(),\n        }\n    }\n}\n\nimpl<'i, T: 'i> fmt::Debug for Slice<'i, T>\nwhere\n    T: fmt::Debug + FlatSerializable<'i> + Clone,\n{\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        f.debug_list().entries(self.iter()).finish()\n    }\n}\n\nimpl<'i, T: 'i> PartialEq for Slice<'i, T>\nwhere\n    T: FlatSerializable<'i> + Clone + PartialEq,\n{\n    fn eq(&self, other: &Self) -> bool {\n        <Iter<'_, '_, T> as Iterator>::eq(self.iter(), other.iter())\n    }\n}\n\nimpl<'i, T: 'i> Eq for Slice<'i, T> where T: FlatSerializable<'i> + Clone + Eq {}\n\n#[derive(Debug)]\npub struct Unflatten<'input, T: 'input> {\n    slice: &'input [u8],\n    _pd: PhantomData<&'input T>,\n}\n\nimpl<'input, T: 'input> Slice<'input, T> {\n    #[allow(clippy::missing_safety_doc)]\n    pub unsafe fn from_bytes(bytes: &'input [u8]) -> Self {\n        Slice::Iter(Unflatten {\n            slice: bytes,\n            _pd: PhantomData,\n        })\n    }\n\n    pub fn len(&self) -> usize\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        match self {\n            Slice::Iter(..) => self.iter().count(),\n            Slice::Slice(s) => s.len(),\n            Slice::Owned(o) => o.len(),\n        }\n    }\n\n    pub fn is_empty(&self) -> bool\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        match self {\n            Slice::Iter(..) => self.iter().count() == 0,\n            Slice::Slice(s) => s.is_empty(),\n            Slice::Owned(o) => o.is_empty(),\n        }\n    }\n\n    pub fn make_owned(&mut self)\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        self.as_owned();\n    }\n\n    pub fn into_vec(self) -> Vec<T::OWNED>\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        match self {\n            Slice::Iter(_) => self.iter().map(|t| t.into_owned()).collect(),\n            Slice::Slice(s) => s.iter().map(|t| t.clone().into_owned()).collect(),\n            Slice::Owned(v) => v.into_iter().map(|t| t.into_owned()).collect(),\n        }\n    }\n\n    pub fn into_owned(self) -> Slice<'static, T::OWNED>\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        Slice::Owned(self.into_vec())\n    }\n\n    pub fn as_owned(&mut self) -> &mut Vec<T>\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        match self {\n            Slice::Iter(_) => {\n                let vec = self.iter().collect();\n                *self = Slice::Owned(vec);\n            }\n            Slice::Slice(s) => {\n                *self = Slice::Owned(s.to_vec());\n            }\n            Slice::Owned(..) => (),\n        }\n        match self {\n            Slice::Owned(vec) => vec,\n            _ => unreachable!(),\n        }\n    }\n\n    pub fn as_slice(&self) -> &[T]\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        match self {\n            Slice::Iter(_) => panic!(\"cannot convert iterator to slice without mutating\"),\n            Slice::Slice(s) => s,\n            Slice::Owned(o) => o,\n        }\n    }\n\n    pub fn slice(&self) -> &'input [T]\n    where\n        T: Clone + FlatSerializable<'input>,\n    {\n        match self {\n            Slice::Slice(s) => s,\n            _ => panic!(\"cannot convert to slice without mutating\"),\n        }\n    }\n}\n\nimpl<'input, T: 'input> Iterator for Unflatten<'input, T>\nwhere\n    T: FlatSerializable<'input>,\n{\n    type Item = T;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        if self.slice.is_empty() {\n            return None;\n        }\n        let (val, rem) = unsafe { <T>::try_ref(self.slice).unwrap() };\n        self.slice = rem;\n        Some(val)\n    }\n}\n\nimpl<'input, T: 'input> From<&'input [T]> for Slice<'input, T> {\n    fn from(val: &'input [T]) -> Self {\n        Self::Slice(val)\n    }\n}\n\nimpl<'input, T: 'input> From<Vec<T>> for Slice<'input, T> {\n    fn from(val: Vec<T>) -> Self {\n        Self::Owned(val)\n    }\n}\n\nimpl<'input, T: 'input> Clone for Slice<'input, T>\nwhere\n    T: Clone,\n{\n    fn clone(&self) -> Self {\n        match self {\n            Slice::Iter(i) => Slice::Iter(*i),\n            Slice::Slice(s) => Slice::Slice(s),\n            Slice::Owned(v) => Slice::Owned(Vec::clone(v)),\n        }\n    }\n}\n\nimpl<'i, T> serde::Serialize for Slice<'i, T>\nwhere\n    T: serde::Serialize + Clone + FlatSerializable<'i>,\n{\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        use serde::ser::SerializeSeq;\n\n        let mut s = serializer.serialize_seq(Some(self.len()))?;\n        for t in self.iter() {\n            s.serialize_element(&t)?\n        }\n        s.end()\n    }\n}\n\nimpl<'de, T> serde::Deserialize<'de> for Slice<'_, T>\nwhere\n    T: serde::Deserialize<'de>,\n{\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        let v = Vec::deserialize(deserializer)?;\n        Ok(Self::Owned(v))\n    }\n}\n\nimpl<'input, T: 'input> Clone for Unflatten<'input, T> {\n    fn clone(&self) -> Self {\n        *self\n    }\n}\n\nimpl<'input, T: 'input> Copy for Unflatten<'input, T> {}\n\n#[doc(hidden)]\npub unsafe trait VariableLen<'input>: Sized {\n    #[allow(clippy::missing_safety_doc)]\n    unsafe fn try_ref(input: &'input [u8], count: usize) -> Result<(Self, &'input [u8]), WrapErr>;\n    #[must_use]\n    #[allow(clippy::missing_safety_doc)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        count: usize,\n        input: &'out mut [MaybeUninit<u8>],\n    ) -> &'out mut [MaybeUninit<u8>];\n    fn num_bytes(&self, count: usize) -> usize;\n}\n\nunsafe impl<'i, T: 'i> VariableLen<'i> for &'i [T]\nwhere\n    T: FlatSerializable<'i>,\n{\n    #[inline(always)]\n    unsafe fn try_ref(input: &'i [u8], count: usize) -> Result<(Self, &'i [u8]), WrapErr> {\n        assert!(<T as FlatSerializable>::TRIVIAL_COPY);\n        let byte_len = T::MIN_LEN * count;\n        if input.len() < byte_len {\n            return Err(WrapErr::NotEnoughBytes(byte_len));\n        }\n        let (bytes, rem) = input.split_at(byte_len);\n        let bytes = bytes.as_ptr();\n        let field = ::std::slice::from_raw_parts(bytes.cast::<T>(), count);\n        debug_assert_eq!(\n            bytes.add(byte_len) as usize,\n            field.as_ptr().add(count) as usize\n        );\n        Ok((field, rem))\n    }\n\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        count: usize,\n        input: &'out mut [MaybeUninit<u8>],\n    ) -> &'out mut [MaybeUninit<u8>] {\n        assert!(<T as FlatSerializable>::TRIVIAL_COPY);\n        if !<T as FlatSerializable>::TRIVIAL_COPY {\n            return fill_slice_from_iter::<T, _, _>(self.iter(), count, input);\n        }\n        let vals = &self[..count];\n        let size = <T>::MIN_LEN * vals.len();\n        let (out, rem) = input.split_at_mut(size);\n        let bytes = vals.as_ptr().cast::<std::mem::MaybeUninit<u8>>();\n        let bytes = std::slice::from_raw_parts(bytes, size);\n        out.copy_from_slice(bytes);\n        rem\n    }\n\n    #[inline(always)]\n    fn num_bytes(&self, count: usize) -> usize {\n        assert!(<T as FlatSerializable>::TRIVIAL_COPY);\n        if !<T as FlatSerializable>::TRIVIAL_COPY {\n            return len_of_iterable::<T, _, _>(self.iter(), count);\n        }\n        ::std::mem::size_of::<T>() * count\n    }\n}\n\nunsafe impl<'i, T: 'i> VariableLen<'i> for Slice<'i, T>\nwhere\n    T: FlatSerializable<'i> + Clone,\n{\n    #[inline(always)]\n    unsafe fn try_ref(input: &'i [u8], count: usize) -> Result<(Self, &'i [u8]), WrapErr> {\n        if T::TRIVIAL_COPY {\n            let (field, rem) = <&[T]>::try_ref(input, count)?;\n            return Ok((Self::Slice(field), rem));\n        }\n        let mut total_len = 0;\n        let mut tmp = input;\n        let mut old_ptr = input.as_ptr() as usize;\n        for _ in 0..count {\n            let (field, rem) = T::try_ref(tmp)?;\n            debug_assert_eq!(rem.as_ptr() as usize - old_ptr, field.num_bytes());\n\n            let additional_len = aligning_len(rem.as_ptr() as _, T::REQUIRED_ALIGNMENT);\n            if rem.len() < additional_len {\n                return Err(WrapErr::NotEnoughBytes(additional_len));\n            }\n\n            let rem = &rem[additional_len..];\n            debug_assert_eq!(rem.as_ptr() as usize % T::REQUIRED_ALIGNMENT, 0);\n\n            let padded_len = rem.as_ptr() as usize - old_ptr;\n\n            old_ptr = rem.as_ptr() as usize;\n            tmp = rem;\n            total_len += padded_len;\n        }\n        let (iter, rem) = input.split_at(total_len);\n        debug_assert_eq!(rem.as_ptr() as usize, tmp.as_ptr() as usize);\n        debug_assert_eq!(rem.len(), tmp.len());\n        Ok((Self::from_bytes(iter), rem))\n    }\n\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        count: usize,\n        input: &'out mut [MaybeUninit<u8>],\n    ) -> &'out mut [MaybeUninit<u8>] {\n        if let (true, Self::Slice(values)) = (T::TRIVIAL_COPY, self) {\n            return <&[T]>::fill_slice(values, count, input);\n        }\n        fill_slice_from_iter(self.iter(), count, input)\n    }\n\n    #[inline(always)]\n    fn num_bytes(&self, count: usize) -> usize {\n        if let (true, Self::Slice(values)) = (T::TRIVIAL_COPY, self) {\n            return <&[T]>::num_bytes(values, count);\n        }\n        len_of_iterable(self.iter(), count)\n    }\n}\n\n#[inline(always)]\nunsafe fn fill_slice_from_iter<\n    'i,\n    'out,\n    T: FlatSerializable<'i>,\n    V: ValOrRef<T>,\n    I: Iterator<Item = V>,\n>(\n    iter: I,\n    count: usize,\n    mut input: &'out mut [MaybeUninit<u8>],\n) -> &'out mut [MaybeUninit<u8>] {\n    let mut filled = 0;\n    for v in iter.take(count) {\n        input = v.to_ref().fill_slice(input);\n        let additional_len = aligning_len(input.as_ptr(), T::REQUIRED_ALIGNMENT);\n        let (addition, rem) = input.split_at_mut(additional_len);\n        addition.copy_from_slice(&[MaybeUninit::new(0); 8][..additional_len]);\n        debug_assert_eq!(rem.as_ptr() as usize % T::REQUIRED_ALIGNMENT, 0);\n        input = rem;\n        filled += 1;\n    }\n    if filled < count {\n        panic!(\"Not enough elements. Expected {count} found {filled}\")\n    }\n    input\n}\n\n#[inline(always)]\nfn len_of_iterable<'i, T: FlatSerializable<'i>, V: ValOrRef<T>, I: Iterator<Item = V>>(\n    iter: I,\n    count: usize,\n) -> usize {\n    let mut filled = 0;\n    let mut len = 0;\n    for v in iter.take(count) {\n        filled += 1;\n        len += v.to_ref().num_bytes();\n        if len % T::REQUIRED_ALIGNMENT != 0 {\n            len += T::REQUIRED_ALIGNMENT - (len % T::REQUIRED_ALIGNMENT);\n        }\n    }\n    if filled < count {\n        panic!(\"Not enough elements. Expected {count} found {filled}\")\n    }\n    len\n}\n\n#[inline(always)]\nfn aligning_len(ptr: *const MaybeUninit<u8>, align: usize) -> usize {\n    let current_ptr = ptr as usize;\n    if current_ptr % align == 0 {\n        return 0;\n    }\n    align - (current_ptr % align)\n}\n\ntrait ValOrRef<T: ?Sized> {\n    fn to_ref(&self) -> &T;\n}\n\nimpl<T: ?Sized> ValOrRef<T> for T {\n    fn to_ref(&self) -> &T {\n        self\n    }\n}\n\nimpl<T: ?Sized> ValOrRef<T> for &T {\n    fn to_ref(&self) -> &T {\n        self\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate as flat_serialize;\n\n    use flat_serialize_macro::{flat_serialize, FlatSerializable};\n\n    flat_serialize! {\n        #[derive(Debug)]\n        struct Basic<'input> {\n            header: u64,\n            data_len: u32,\n            array: [u16; 3],\n            data: [u8; self.data_len],\n            data2: [[u8; 2]; self.data_len / 3],\n        }\n    }\n\n    #[test]\n    fn basic() {\n        use crate::{FlatSerializable, Slice, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&33u64.to_ne_bytes());\n        bytes.extend_from_slice(&6u32.to_ne_bytes());\n        bytes.extend_from_slice(&202u16.to_ne_bytes());\n        bytes.extend_from_slice(&404u16.to_ne_bytes());\n        bytes.extend_from_slice(&555u16.to_ne_bytes());\n        bytes.extend_from_slice(&[1, 3, 5, 7, 9, 11]);\n        bytes.extend_from_slice(&[4, 4, 95, 99]);\n        let (\n            Basic {\n                header,\n                data_len,\n                data,\n                data2,\n                array,\n            },\n            rem,\n        ) = unsafe { Basic::try_ref(&bytes).unwrap() };\n        assert_eq!(\n            (header, data_len, array, &data, &data2, rem),\n            (\n                33,\n                6,\n                [202, 404, 555],\n                &Slice::Slice(&[1, 3, 5, 7, 9, 11][..]),\n                &Slice::Slice(&[[4, 4], [95, 99]]),\n                &[][..]\n            )\n        );\n\n        let mut output = vec![];\n        Basic {\n            header,\n            data_len,\n            data: data.clone(),\n            data2: data2.clone(),\n            array,\n        }\n        .fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        let debug = format!(\n            \"{:?}\",\n            Basic {\n                header,\n                data_len,\n                data,\n                data2,\n                array\n            }\n        );\n        assert_eq!(debug, \"Basic { header: 33, data_len: 6, array: [202, 404, 555], data: [1, 3, 5, 7, 9, 11], data2: [[4, 4], [95, 99]] }\");\n\n        assert_eq!(Basic::MIN_LEN, 18);\n        assert_eq!(Basic::REQUIRED_ALIGNMENT, 8);\n        assert_eq!(Basic::MAX_PROVIDED_ALIGNMENT, Some(1));\n        assert_eq!(Basic::TRIVIAL_COPY, false);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Basic::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    #[test]\n    #[should_panic(expected = \"range end index 5 out of range for slice of length 1\")]\n    fn bad_len1() {\n        use crate::{FlatSerializable, Slice};\n        let mut output = vec![];\n        Basic {\n            header: 1,\n            data_len: 5,\n            array: [0; 3],\n            data: Slice::Slice(&[1]),\n            data2: Slice::Slice(&[[2, 2]]),\n        }\n        .fill_vec(&mut output);\n    }\n\n    #[test]\n    #[should_panic(expected = \"range end index 1 out of range for slice of length 0\")]\n    fn bad_len2() {\n        use crate::{FlatSerializable, Slice};\n        let mut output = vec![];\n        Basic {\n            header: 1,\n            data_len: 5,\n            array: [0; 3],\n            data: Slice::Slice(&[1, 2, 3, 4, 5]),\n            data2: Slice::Slice(&[]),\n        }\n        .fill_vec(&mut output);\n    }\n\n    flat_serialize! {\n        #[derive(Debug, PartialEq, Eq)]\n        struct Optional {\n            header: u64,\n            optional_field: u32 if self.header != 1,\n            non_optional_field: u16,\n        }\n    }\n\n    const _TEST_NO_VARIABLE_LEN_NO_LIFETIME: Optional = Optional {\n        header: 0,\n        optional_field: None,\n        non_optional_field: 0,\n    };\n\n    #[test]\n    fn optional_present() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&101010101u64.to_ne_bytes());\n        bytes.extend_from_slice(&30u32.to_ne_bytes());\n        bytes.extend_from_slice(&6u16.to_ne_bytes());\n        let (\n            Optional {\n                header,\n                optional_field,\n                non_optional_field,\n            },\n            rem,\n        ) = unsafe { Optional::try_ref(&bytes).unwrap() };\n        assert_eq!(\n            (header, optional_field, non_optional_field, rem),\n            (101010101, Some(30), 6, &[][..])\n        );\n\n        let mut output = vec![];\n        Optional {\n            header,\n            optional_field,\n            non_optional_field,\n        }\n        .fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Optional::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n\n        assert_eq!(Optional::MIN_LEN, 10);\n        assert_eq!(Optional::REQUIRED_ALIGNMENT, 8);\n        assert_eq!(Optional::MAX_PROVIDED_ALIGNMENT, Some(2));\n        assert_eq!(Optional::TRIVIAL_COPY, false);\n    }\n\n    #[test]\n    fn optional_absent() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&1u64.to_ne_bytes());\n        bytes.extend_from_slice(&7u16.to_ne_bytes());\n        let (\n            Optional {\n                header,\n                optional_field,\n                non_optional_field,\n            },\n            rem,\n        ) = unsafe { Optional::try_ref(&bytes).unwrap() };\n        assert_eq!(\n            (header, optional_field, non_optional_field, rem),\n            (1, None, 7, &[][..])\n        );\n\n        let mut output = vec![];\n        Optional {\n            header,\n            optional_field,\n            non_optional_field,\n        }\n        .fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Optional::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    flat_serialize! {\n        #[derive(Debug)]\n        struct Nested<'a> {\n            prefix: u64,\n            basic: Basic<'a>,\n        }\n    }\n\n    #[test]\n    fn nested() {\n        use crate::{FlatSerializable, Slice, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&101010101u64.to_ne_bytes());\n        bytes.extend_from_slice(&33u64.to_ne_bytes());\n        bytes.extend_from_slice(&6u32.to_ne_bytes());\n        bytes.extend_from_slice(&202u16.to_ne_bytes());\n        bytes.extend_from_slice(&404u16.to_ne_bytes());\n        bytes.extend_from_slice(&555u16.to_ne_bytes());\n        bytes.extend_from_slice(&[1, 3, 5, 7, 9, 11]);\n        bytes.extend_from_slice(&[3, 0, 104, 2]);\n        let (\n            Nested {\n                prefix,\n                basic:\n                    Basic {\n                        header,\n                        data_len,\n                        array,\n                        data,\n                        data2,\n                    },\n            },\n            rem,\n        ) = unsafe { Nested::try_ref(&bytes).unwrap() };\n        assert_eq!(\n            (prefix, header, data_len, array, &data, &data2, rem),\n            (\n                101010101,\n                33,\n                6,\n                [202, 404, 555],\n                &Slice::Slice(&[1, 3, 5, 7, 9, 11][..]),\n                &Slice::Slice(&[[3, 0], [104, 2]]),\n                &[][..]\n            )\n        );\n\n        let mut output = vec![];\n        Nested {\n            prefix,\n            basic: Basic {\n                header,\n                data_len,\n                data,\n                data2,\n                array,\n            },\n        }\n        .fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Nested::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    flat_serialize! {\n        #[derive(Debug)]\n        struct NestedOptional {\n            present: u64,\n            val: Optional if self.present > 2,\n        }\n    }\n\n    #[test]\n    fn nested_optional() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&3u64.to_ne_bytes());\n        {\n            bytes.extend_from_slice(&0u64.to_ne_bytes());\n            bytes.extend_from_slice(&111111111u32.to_ne_bytes());\n            bytes.extend_from_slice(&0xf00fu16.to_ne_bytes());\n            bytes.extend_from_slice(&[77; 2]);\n        }\n\n        let (NestedOptional { present, val }, rem) =\n            unsafe { NestedOptional::try_ref(&bytes).unwrap() };\n\n        assert_eq!(\n            (present, &val, rem),\n            (\n                3,\n                &Some(Optional {\n                    header: 0,\n                    optional_field: Some(111111111),\n                    non_optional_field: 0xf00f,\n                }),\n                &[77; 2][..],\n            )\n        );\n\n        let mut output = vec![];\n        NestedOptional { present, val }.fill_vec(&mut output);\n        assert_eq!(output, &bytes[..bytes.len() - 2]);\n\n        for i in 0..bytes.len() - 3 {\n            let res = unsafe { NestedOptional::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n\n        assert_eq!(NestedOptional::MIN_LEN, 8);\n        assert_eq!(NestedOptional::REQUIRED_ALIGNMENT, 8);\n        assert_eq!(NestedOptional::MAX_PROVIDED_ALIGNMENT, Some(2));\n        assert_eq!(NestedOptional::TRIVIAL_COPY, false);\n    }\n\n    flat_serialize! {\n        #[derive(Debug)]\n        struct NestedSlice<'b> {\n            num_vals: u64,\n            // #[flat_serialize::flatten]\n            vals: [Optional; self.num_vals],\n        }\n    }\n\n    #[test]\n    fn nested_slice() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&3u64.to_ne_bytes());\n        {\n            bytes.extend_from_slice(&101010101u64.to_ne_bytes());\n            bytes.extend_from_slice(&30u32.to_ne_bytes());\n            bytes.extend_from_slice(&6u16.to_ne_bytes());\n            bytes.extend_from_slice(&[0; 2]);\n        }\n        {\n            bytes.extend_from_slice(&1u64.to_ne_bytes());\n            bytes.extend_from_slice(&7u16.to_ne_bytes());\n            bytes.extend_from_slice(&[0; 6]);\n        }\n        {\n            bytes.extend_from_slice(&0u64.to_ne_bytes());\n            bytes.extend_from_slice(&111111111u32.to_ne_bytes());\n            bytes.extend_from_slice(&0xf00fu16.to_ne_bytes());\n            bytes.extend_from_slice(&[0; 2]);\n        }\n\n        let (NestedSlice { num_vals, vals }, rem) =\n            unsafe { NestedSlice::try_ref(&bytes).unwrap() };\n        let vals_vec: Vec<_> = vals.iter().collect();\n        assert_eq!(\n            (num_vals, &*vals_vec, rem),\n            (\n                3,\n                &[\n                    Optional {\n                        header: 101010101,\n                        optional_field: Some(30),\n                        non_optional_field: 6,\n                    },\n                    Optional {\n                        header: 1,\n                        optional_field: None,\n                        non_optional_field: 7,\n                    },\n                    Optional {\n                        header: 0,\n                        optional_field: Some(111111111),\n                        non_optional_field: 0xf00f,\n                    },\n                ][..],\n                &[][..],\n            )\n        );\n\n        let mut output = vec![];\n        NestedSlice { num_vals, vals }.fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { NestedSlice::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n\n        assert_eq!(NestedSlice::MIN_LEN, 8);\n        assert_eq!(NestedSlice::REQUIRED_ALIGNMENT, 8);\n        assert_eq!(NestedSlice::MAX_PROVIDED_ALIGNMENT, Some(8));\n        assert_eq!(NestedSlice::TRIVIAL_COPY, false);\n    }\n\n    flat_serialize! {\n        #[derive(Debug, PartialEq, Eq)]\n        enum BasicEnum<'input> {\n            k: u64,\n            First: 2 {\n                data_len: u32,\n                data: [u8; self.data_len],\n            },\n            Fixed: 3 {\n                array: [u16; 3],\n            },\n        }\n    }\n\n    #[test]\n    fn basic_enum1() {\n        use crate::{FlatSerializable, Slice, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&2u64.to_ne_bytes());\n        bytes.extend_from_slice(&6u32.to_ne_bytes());\n        bytes.extend_from_slice(&[1, 3, 5, 7, 9, 11]);\n        let (data_len, data, rem) = match unsafe { BasicEnum::try_ref(&bytes).unwrap() } {\n            (BasicEnum::First { data_len, data }, rem) => (data_len, data, rem),\n            _ => unreachable!(),\n        };\n        assert_eq!(\n            (data_len, &data, rem),\n            (6, &Slice::Slice(&[1, 3, 5, 7, 9, 11][..]), &[][..])\n        );\n\n        let mut output = vec![];\n        BasicEnum::First { data_len, data }.fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { BasicEnum::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    #[test]\n    fn basic_enum2() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&3u64.to_ne_bytes());\n        bytes.extend_from_slice(&3u16.to_ne_bytes());\n        bytes.extend_from_slice(&6u16.to_ne_bytes());\n        bytes.extend_from_slice(&9u16.to_ne_bytes());\n        bytes.extend_from_slice(&[7]);\n        let (array, rem) = match unsafe { BasicEnum::try_ref(&bytes).unwrap() } {\n            (BasicEnum::Fixed { array }, rem) => (array, rem),\n            _ => unreachable!(),\n        };\n        assert_eq!((array, rem), ([3, 6, 9], &[7][..]));\n\n        let (array, rem) = match unsafe { BasicEnum::try_ref(&bytes).unwrap() } {\n            (BasicEnum::Fixed { array }, rem) => (array, rem),\n            _ => unreachable!(),\n        };\n        assert_eq!((array, rem), ([3, 6, 9], &[7][..]));\n\n        let mut output = vec![];\n        BasicEnum::Fixed { array }.fill_vec(&mut output);\n        assert_eq!(output, &bytes[..bytes.len() - 1]);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { BasicEnum::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    flat_serialize! {\n        #[derive(Debug)]\n        enum PaddedEnum<'input> {\n            k: u8,\n            First: 2 {\n                padding: [u8; 3],\n                data_len: u32,\n                data: [u8; self.data_len],\n            },\n            Fixed: 3 {\n                padding: u8,\n                array: [u16; 3],\n            },\n        }\n    }\n\n    #[test]\n    fn padded_enum1() {\n        use crate::{FlatSerializable, Slice, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&2u8.to_ne_bytes());\n        bytes.extend_from_slice(&[0xf, 0xf, 0xf]);\n        bytes.extend_from_slice(&6u32.to_ne_bytes());\n        bytes.extend_from_slice(&[1, 3, 5, 7, 9, 11]);\n        let (padding, data_len, data, rem) = match unsafe { PaddedEnum::try_ref(&bytes).unwrap() } {\n            (\n                PaddedEnum::First {\n                    padding,\n                    data_len,\n                    data,\n                },\n                rem,\n            ) => (padding, data_len, data, rem),\n            _ => unreachable!(),\n        };\n        assert_eq!(\n            (padding, data_len, &data, rem),\n            (\n                [0xf, 0xf, 0xf],\n                6,\n                &Slice::Slice(&[1, 3, 5, 7, 9, 11][..]),\n                &[][..]\n            )\n        );\n\n        let mut output = vec![];\n        PaddedEnum::First {\n            padding,\n            data_len,\n            data,\n        }\n        .fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { PaddedEnum::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    #[test]\n    fn padded_enum2() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&3u8.to_ne_bytes());\n        bytes.extend_from_slice(&[0]);\n        bytes.extend_from_slice(&3u16.to_ne_bytes());\n        bytes.extend_from_slice(&6u16.to_ne_bytes());\n        bytes.extend_from_slice(&9u16.to_ne_bytes());\n        bytes.extend_from_slice(&[7]);\n        let (padding, array, rem) = match unsafe { PaddedEnum::try_ref(&bytes).unwrap() } {\n            (PaddedEnum::Fixed { padding, array }, rem) => (padding, array, rem),\n            _ => unreachable!(),\n        };\n        assert_eq!((padding, array, rem), (0, [3, 6, 9], &[7][..]));\n\n        let (padding, array, rem) = match unsafe { PaddedEnum::try_ref(&bytes).unwrap() } {\n            (PaddedEnum::Fixed { padding, array }, rem) => (padding, array, rem),\n            _ => unreachable!(),\n        };\n        assert_eq!((padding, array, rem), (0, [3, 6, 9], &[7][..]));\n\n        let mut output = vec![];\n        PaddedEnum::Fixed { padding, array }.fill_vec(&mut output);\n        assert_eq!(output, &bytes[..bytes.len() - 1]);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { PaddedEnum::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    flat_serialize! {\n        #[derive(Debug)]\n        struct ManyEnum<'input> {\n            count: u64,\n            enums: [BasicEnum<'input>; self.count],\n        }\n    }\n\n    #[test]\n    fn many_enum() {\n        use crate::{FlatSerializable, Slice, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&4u64.to_ne_bytes());\n        {\n            bytes.extend_from_slice(&2u64.to_ne_bytes());\n            bytes.extend_from_slice(&6u32.to_ne_bytes());\n            bytes.extend_from_slice(&[1, 3, 5, 7, 9, 11]);\n            while bytes.len() % 8 != 0 {\n                bytes.push(0)\n            }\n        }\n        {\n            bytes.extend_from_slice(&3u64.to_ne_bytes());\n            bytes.extend_from_slice(&3u16.to_ne_bytes());\n            bytes.extend_from_slice(&6u16.to_ne_bytes());\n            bytes.extend_from_slice(&9u16.to_ne_bytes());\n            while bytes.len() % 8 != 0 {\n                bytes.push(0)\n            }\n        }\n        {\n            bytes.extend_from_slice(&2u64.to_ne_bytes());\n            bytes.extend_from_slice(&1u32.to_ne_bytes());\n            bytes.extend_from_slice(&[44u8]);\n            while bytes.len() % 8 != 0 {\n                bytes.push(0)\n            }\n        }\n        {\n            bytes.extend_from_slice(&2u64.to_ne_bytes());\n            bytes.extend_from_slice(&2u32.to_ne_bytes());\n            bytes.extend_from_slice(&[89u8, 123u8]);\n            while bytes.len() % 8 != 0 {\n                bytes.push(0)\n            }\n        }\n\n        let (ManyEnum { count, enums }, rem) = unsafe { ManyEnum::try_ref(&bytes).unwrap() };\n        assert_eq!((count, rem), (4, &[][..]));\n        let enums_vec: Vec<_> = enums.iter().collect();\n        assert_eq!(\n            &*enums_vec,\n            &[\n                BasicEnum::First {\n                    data_len: 6,\n                    data: Slice::Slice(&[1, 3, 5, 7, 9, 11])\n                },\n                BasicEnum::Fixed { array: [3, 6, 9] },\n                BasicEnum::First {\n                    data_len: 1,\n                    data: Slice::Slice(&[44u8])\n                },\n                BasicEnum::First {\n                    data_len: 2,\n                    data: Slice::Slice(&[89u8, 123])\n                },\n            ]\n        );\n\n        let mut output = vec![];\n        ManyEnum { count, enums }.fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { ManyEnum::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    macro_rules! sub_macro {\n        (\n            $(#[$attrs: meta])?\n            struct $name: ident {\n                $($field:ident : $typ: tt),*\n                $(,)?\n            }\n        ) => {\n            flat_serialize_macro::flat_serialize! {\n                $(#[$attrs])?\n                struct $name {\n                    $($field: $typ),*\n                }\n            }\n        }\n    }\n\n    // test that sub_macros provide correct compilation\n    sub_macro! {\n        #[derive(Debug)]\n        struct InMacro {\n            a: u32,\n            padding: [u8; 4], // with this commented out, the error should be on b\n            b: f64,\n        }\n    }\n\n    #[test]\n    fn test_no_refrence() {\n        flat_serialize! {\n            struct NoLifetime {\n                val: i64,\n            }\n        }\n\n        let _: NoLifetime = NoLifetime { val: 3 };\n\n        flat_serialize! {\n            struct NestedNoLifetime {\n                nested: NoLifetime,\n            }\n        }\n\n        let _: NestedNoLifetime = NestedNoLifetime {\n            nested: NoLifetime { val: 3 },\n        };\n\n        flat_serialize! {\n            enum ENoLifetime {\n                tag: i64,\n                Variant: 1 {\n                    val: i64,\n                },\n            }\n        }\n\n        let _: ENoLifetime = ENoLifetime::Variant { val: 2 };\n\n        flat_serialize! {\n            enum NestedENoLifetime {\n                tag: i64,\n                Variant: 2 {\n                    val: ENoLifetime,\n                },\n            }\n        }\n\n        let _: NestedENoLifetime = NestedENoLifetime::Variant {\n            val: ENoLifetime::Variant { val: 2 },\n        };\n    }\n\n    macro_rules! check_size_align {\n        (struct $($dec_life:lifetime)? {\n            $( $(#[$attrs: meta])*  $field:ident : $typ: tt $(<$life:lifetime>)?),*\n            $(,)?\n        }\n            len: $min_len: expr,\n            align: $required_alignment: expr,\n            max: $max_provided_alignment: expr $(,)?\n        ) => {\n            {\n                flat_serialize!{\n                    struct SizeAlignTest $(<$dec_life>)? {\n                        $($(#[$attrs])* $field: $typ $(<$life>)?),*\n                    }\n                };\n                assert_eq!(<SizeAlignTest as crate::FlatSerializable>::MIN_LEN, $min_len, \"length\");\n                assert_eq!(<SizeAlignTest as crate::FlatSerializable>::REQUIRED_ALIGNMENT, $required_alignment, \"required\");\n                assert_eq!(<SizeAlignTest as crate::FlatSerializable>::MAX_PROVIDED_ALIGNMENT, $max_provided_alignment, \"max provided\");\n                assert_eq!(<SizeAlignTest as crate::FlatSerializable>::TRIVIAL_COPY, false, \"trivial copy\");\n            }\n        }\n    }\n\n    #[test]\n    fn test_size_align_struct() {\n        check_size_align!(\n            struct {\n                f: u8,\n            }\n            len: 1,\n            align: 1,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                f: u16,\n            }\n            len: 2,\n            align: 2,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                f: u32,\n            }\n            len: 4,\n            align: 4,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                f: u64,\n            }\n            len: 8,\n            align: 8,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                a: u64,\n                b: u32,\n                c: u16,\n            }\n            len: 8 + 4 + 2,\n            align: 8,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                a: u32,\n                b: u32,\n                c: u32,\n            }\n            len: 4 + 4 + 4,\n            align: 4,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                a: [u32; 3],\n            }\n            len: 4 * 3,\n            align: 4,\n            max: None,\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: u32,\n                b: [u16; self.a],\n            }\n            len: 4,\n            align: 4,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: u32,\n                b: [u32; self.a],\n            }\n            len: 4,\n            align: 4,\n            max: Some(4),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: u32,\n                b: [u32; self.a],\n                c: u32,\n            }\n            len: 4 + 4,\n            align: 4,\n            max: Some(4),\n        );\n\n        flat_serialize! {\n            struct NestedA {\n                a: u32,\n                b: u16,\n            }\n        }\n\n        check_size_align!(\n            struct {\n                a: u32,\n                b: NestedA,\n            }\n            len: 4 + (4 + 2),\n            align: 4,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                a: u64,\n                b: NestedA,\n            }\n            len: 8 + (4 + 2),\n            align: 8,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                a: u64,\n                b: NestedA,\n                c: u8\n            }\n            len: 8 + (4 + 2) + 1,\n            align: 8,\n            max: None,\n        );\n\n        check_size_align!(\n            struct {\n                a: NestedA,\n                b: u8,\n                c: u8,\n                f: NestedA,\n            }\n            len: (4 + 2) + 1 + 1 + (4 + 2),\n            align: 4,\n            max: None,\n        );\n\n        flat_serialize! {\n            struct NestedB<'input> {\n                a: u32,\n                b: [u16; self.a],\n            }\n        }\n\n        check_size_align!(\n            struct 'a {\n                a: u32,\n                b: NestedB<'a>,\n            }\n            len: 4 + (4),\n            align: 4,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: u64,\n                b: NestedB<'a>,\n            }\n            len: 8 + (4),\n            align: 8,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: u64,\n                b: NestedB<'a>,\n                c: u8\n            }\n            len: 8 + (4) + 1,\n            align: 8,\n            max: Some(1),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: u8,\n                b: u8,\n                c: u8,\n                d: u8,\n                e: NestedB<'a>,\n            }\n            len: 4 + (4),\n            align: 4,\n            max: Some(2),\n        );\n    }\n\n    #[test]\n    fn test_size_align_enum() {\n        flat_serialize! {\n            enum EnumA {\n                tag: u32,\n                A: 1 {\n                    a: u32,\n                },\n                B: 2 {\n                    a: u16,\n                },\n            }\n        }\n\n        check_size_align!(\n            struct {\n                a: EnumA,\n            }\n            len: (4 + 2),\n            align: 4,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct {\n                a: EnumA,\n                b: u16,\n            }\n            len: (4 + 2) + 2,\n            align: 4,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct {\n                b: u64,\n                a: EnumA,\n            }\n            len: 8 + (4 + 2),\n            align: 8,\n            max: Some(2),\n        );\n\n        flat_serialize! {\n            enum EnumB {\n                tag: u32,\n                A: 1 {\n                    a: [u8; 5],\n                },\n                B: 2 {\n                    a: u16,\n                },\n            }\n        }\n\n        check_size_align!(\n            struct {\n                a: EnumB,\n            }\n            len: (4 + 2),\n            align: 4,\n            max: Some(1),\n        );\n\n        check_size_align!(\n            struct {\n                b: u64,\n                a: EnumB,\n            }\n            len: 8 + (4 + 2),\n            align: 8,\n            max: Some(1),\n        );\n\n        flat_serialize! {\n            enum EnumC<'input> {\n                tag: u64,\n                A: 1 {\n                    a: u64,\n                },\n                B: 2 {\n                    a: u16,\n                    b: [u16; self.a],\n                },\n            }\n        }\n\n        check_size_align!(\n            struct 'a {\n                a: EnumC<'a>,\n            }\n            len: (8 + 2),\n            align: 8,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: EnumC<'a>,\n                b: u16,\n            }\n            len: (8 + 2) + 2,\n            align: 8,\n            max: Some(2),\n        );\n\n        check_size_align!(\n            struct 'a {\n                b: u64,\n                a: EnumC<'a>,\n            }\n            len: 8 + (8 + 2),\n            align: 8,\n            max: Some(2),\n        );\n\n        flat_serialize! {\n            enum EnumD<'input> {\n                tag: u32,\n                A: 1 {\n                    a: u16,\n                },\n                B: 2 {\n                    a: u32,\n                    b: [u8; self.a],\n                },\n            }\n        }\n\n        check_size_align!(\n            struct 'a {\n                a: EnumD<'a>,\n            }\n            len: (4 + 2),\n            align: 4,\n            max: Some(1),\n        );\n\n        check_size_align!(\n            struct 'a {\n                a: EnumD<'a>,\n                b: u8,\n            }\n            len: (4 + 2) + 1,\n            align: 4,\n            max: Some(1),\n        );\n\n        check_size_align!(\n            struct 'a {\n                b: u64,\n                a: EnumD<'a>,\n            }\n            len: 8 + (4 + 2),\n            align: 8,\n            max: Some(1),\n        );\n    }\n\n    #[derive(FlatSerializable)]\n    #[allow(dead_code)]\n    #[derive(Debug)]\n    #[repr(C)]\n    struct Foo {\n        a: i32,\n        b: i32,\n    }\n\n    const _: () = {\n        fn check_flat_serializable_impl<'a, T: crate::FlatSerializable<'a>>() {}\n        let _ = check_flat_serializable_impl::<Foo>;\n        let _ = check_flat_serializable_impl::<[Foo; 2]>;\n    };\n\n    #[test]\n    fn foo() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&33i32.to_ne_bytes());\n        bytes.extend_from_slice(&100000001i32.to_ne_bytes());\n\n        let (Foo { a, b }, rem) = unsafe { Foo::try_ref(&bytes).unwrap() };\n        assert_eq!((a, b, rem), (33, 100000001, &[][..]),);\n\n        let mut output = vec![];\n        Foo { a, b }.fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        assert_eq!(Foo::MIN_LEN, 8);\n        assert_eq!(Foo::REQUIRED_ALIGNMENT, 4);\n        assert_eq!(Foo::MAX_PROVIDED_ALIGNMENT, None);\n        assert_eq!(Foo::TRIVIAL_COPY, true);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Foo::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    #[derive(FlatSerializable)]\n    #[allow(dead_code)]\n    #[repr(u16)]\n    #[derive(Debug, Copy, Clone)]\n    enum Bar {\n        A = 0,\n        B = 1111,\n    }\n\n    const _: () = {\n        fn check_flat_serializable_impl<'a, T: crate::FlatSerializable<'a>>() {}\n        let _ = check_flat_serializable_impl::<Bar>;\n        let _ = check_flat_serializable_impl::<[Bar; 2]>;\n    };\n\n    #[test]\n    fn fs_enum_a() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&0u16.to_ne_bytes());\n\n        let (val, rem) = unsafe { Bar::try_ref(&bytes).unwrap() };\n        assert_eq!((val as u16, rem), (Bar::A as u16, &[][..]));\n\n        let mut output = vec![];\n        val.fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        assert_eq!(Bar::MIN_LEN, 2);\n        assert_eq!(Bar::REQUIRED_ALIGNMENT, 2);\n        assert_eq!(Bar::MAX_PROVIDED_ALIGNMENT, None);\n        assert_eq!(Bar::TRIVIAL_COPY, true);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Bar::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    #[test]\n    fn fs_enum_b() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&1111u16.to_ne_bytes());\n\n        let (val, rem) = unsafe { Bar::try_ref(&bytes).unwrap() };\n        assert_eq!((val as u16, rem), (Bar::B as u16, &[][..]));\n\n        let mut output = vec![];\n        val.fill_vec(&mut output);\n        assert_eq!(output, bytes);\n\n        for i in 0..bytes.len() - 1 {\n            let res = unsafe { Bar::try_ref(&bytes[..i]) };\n            assert!(matches!(res, Err(WrapErr::NotEnoughBytes(..))), \"{:?}\", res);\n        }\n    }\n\n    #[test]\n    fn fs_enum_non() {\n        use crate::{FlatSerializable, WrapErr};\n        let mut bytes = Vec::new();\n        bytes.extend_from_slice(&1u16.to_ne_bytes());\n\n        let res = unsafe { Bar::try_ref(&bytes) };\n        assert!(matches!(res, Err(WrapErr::InvalidTag(0))));\n    }\n}\n"
  },
  {
    "path": "crates/flat_serialize/flat_serialize_macro/Cargo.toml",
    "content": "[package]\nname = \"flat_serialize_macro\"\nversion = \"0.1.0\"\nauthors = [\"Joshua Lockerman\"]\nedition = \"2021\"\n\n[lib]\nproc-macro = true\n\n[dependencies]\nsyn = {version=\"1.0\", features=[\"extra-traits\", \"visit\", \"visit-mut\", \"full\"]}\nquote = \"1.0\"\nproc-macro2 = \"1.0\"\n\n[features]\ndefault = []\nprint-generated = []\n"
  },
  {
    "path": "crates/flat_serialize/flat_serialize_macro/src/lib.rs",
    "content": "use proc_macro::TokenStream;\n\nuse proc_macro2::TokenStream as TokenStream2;\n\nuse quote::{quote, quote_spanned};\n\nuse syn::{\n    parse_macro_input, punctuated::Punctuated, spanned::Spanned, visit_mut::VisitMut, Attribute,\n    Expr, Field, Ident, Lifetime, Token,\n};\n\nmod parser;\n\n#[proc_macro]\npub fn flat_serialize(input: TokenStream) -> TokenStream {\n    // Parse the input tokens into a syntax tree\n    let input = parse_macro_input!(input as FlatSerialize);\n    let expanded = match input {\n        FlatSerialize::Struct(input) => flat_serialize_struct(input),\n        FlatSerialize::Enum(input) => flat_serialize_enum(input),\n    };\n    if cfg!(feature = \"print-generated\") {\n        println!(\"{expanded}\");\n    }\n    expanded.into()\n}\n\n#[allow(clippy::large_enum_variant)] // only one of these are created, and it's on the stack\nenum FlatSerialize {\n    Enum(FlatSerializeEnum),\n    Struct(FlatSerializeStruct),\n}\n\n/// a `flat_serialize`d enum e.g.\n/// ```skip\n/// flat_serialize! {\n///     enum BasicEnum {\n///         k: u8,\n///         First: 2 {\n///             data_len: usize,\n///             data: [u8; self.data_len],\n///         },\n///         Fixed: 3 {\n///             array: [u16; 3],\n///         },\n///     }\n/// }\n/// ```\n/// the body of the enum variants must be the a valid FlatSerializeStruct body\nstruct FlatSerializeEnum {\n    per_field_attrs: Vec<PerFieldsAttr>,\n    attrs: Vec<Attribute>,\n    ident: Ident,\n    lifetime: Option<Lifetime>,\n    tag: FlatSerializeField,\n    variants: Punctuated<FlatSerializeVariant, Token![,]>,\n}\n\nstruct FlatSerializeVariant {\n    tag_val: Expr,\n    body: FlatSerializeStruct,\n}\n\n/// a `flat_serialize`d struct e.g.\n/// ```skip\n/// flat_serialize! {\n///     #[derive(Debug)]\n///     struct Basic {\n///         header: u64,\n///         data_len: u32,\n///         array: [u16; 3],\n///         data: [u8; self.data_len],\n///         data2: [u8; self.data_len / 2],\n///     }\n/// }\n/// ```\n/// the syntax is the same as a regular struct, except that it allows\n/// `self` expressions in the length of arrays; these will be represented as\n/// variable-length fields. We also interpret\n/// `#[flat_serialize::field_attr(fixed = \"#[foo]\", variable = \"#[bar]\"))]` as\n/// applying the attribute `#[foo]` to every fixed-length field of the struct,\n/// and `#[bar]` to every variable-length field. e.g.\n/// ```skip\n/// flat_serialize! {\n///     #[flat_serialize::field_attr(fixed = \"#[foo]\", variable = \"#[bar]\"))]`\n///     struct Struct {\n///         a: i32,\n///         b: i32,\n///         c: [u16; self.a]\n///         d: [u8; self.a]\n///     }\n/// ```\n/// is equivalent to\n/// ```skip\n/// flat_serialize! {\n///     struct Struct {\n///         #[foo]\n///         a: i32,\n///         #[foo]\n///         b: i32,\n///         #[bar]\n///         c: [u16; self.a]\n///         #[bar]\n///         d: [u8; self.a]\n///     }\n/// ```\n/// This can be useful when generating flat_serialize structs from a macro\nstruct FlatSerializeStruct {\n    per_field_attrs: Vec<PerFieldsAttr>,\n    attrs: Vec<Attribute>,\n    ident: Ident,\n    lifetime: Option<Lifetime>,\n    fields: Punctuated<FlatSerializeField, Token![,]>,\n}\nstruct FlatSerializeField {\n    field: Field,\n    ty_without_lifetime: Option<TokenStream2>,\n    // TODO is this mutually exclusive with `flatten` above? Should we make an\n    // enum to select between them?\n    length_info: Option<VariableLenFieldInfo>,\n}\n\n/// a `#[flat_serialize::field_attr(fixed = \"#[foo]\", variable = \"#[bar]\"))]`\n/// attribute. The inner attribute(s) will be applied to each relevant field.\nstruct PerFieldsAttr {\n    fixed: Attribute,\n    variable: Option<Attribute>,\n}\n\n/// how to find the length of a variable-length or optional field.\nstruct VariableLenFieldInfo {\n    ty: syn::Type,\n    ty_without_lifetime: Option<TokenStream2>,\n    len_expr: syn::Expr,\n    // is an optional field instead of a general varlen field, len_expr should\n    // eval to a boolean\n    is_optional: bool,\n}\n\n#[allow(clippy::redundant_clone)] // triggers incorrectly\nfn flat_serialize_struct(input: FlatSerializeStruct) -> TokenStream2 {\n    let ident = input.ident.clone();\n\n    let ref_def = {\n        let alignment_check = input.alignment_check(quote!(0), quote!(8));\n        let trait_check = input.fn_trait_check();\n        let required_alignment = input.fn_required_alignment();\n        let max_provided_alignment = input.fn_max_provided_alignment();\n        let min_len = input.fn_min_len();\n\n        // if we ever want to force #[repr(C)] we can use this code to derive\n        // TRIVIAL_COPY from the struct fields\n        let _const_len = input.fields.iter().map(|f| {\n            if f.length_info.is_some() {\n                quote!(false)\n            } else {\n                let ty = &f.ty;\n                quote!( <#ty as flat_serialize::FlatSerializable>::TRIVIAL_COPY )\n            }\n        });\n\n        let lifetime = input.lifetime.as_ref().map(|lifetime| {\n            quote! { #lifetime }\n        });\n        let try_ref = input.fn_try_ref(lifetime.as_ref());\n        let fill_slice = input.fn_fill_slice();\n        let len = input.fn_len();\n        let field_names = input.fields.iter().map(|f| &f.ident);\n        let field_names1 = field_names.clone();\n        let make_owned = input.fields.iter().map(|f| f.make_owned());\n        let into_owned = input.fields.iter().map(|f| f.into_owned());\n        let fields = input\n            .fields\n            .iter()\n            .map(|f| f.declaration(true, lifetime.as_ref(), input.per_field_attrs.iter()));\n        let lifetime_args = input.lifetime.as_ref().map(|lifetime| {\n            quote! { <#lifetime> }\n        });\n        let ref_liftime = lifetime_args.clone().unwrap_or_else(|| quote! { <'a> });\n        let rl = lifetime.clone().unwrap_or_else(|| quote! { 'a });\n        let owned_lifetime = if lifetime_args.is_some() {\n            Some(quote!( <'static> ))\n        } else {\n            None\n        };\n\n        let attrs = &*input.attrs;\n\n        quote! {\n            #[derive(Clone)]\n            #(#attrs)*\n            pub struct #ident #lifetime_args {\n                #(#fields)*\n            }\n\n            // alignment assertions\n            #[allow(unused_assignments)]\n            const _: () = #alignment_check;\n\n            #trait_check\n\n\n            unsafe impl #ref_liftime flat_serialize::FlatSerializable #ref_liftime for #ident #lifetime_args {\n                #required_alignment\n\n                #max_provided_alignment\n\n                #min_len\n\n                // cannot be TRIVIAL_COPY unless the struct is #[repr(C)]\n                const TRIVIAL_COPY: bool = false;\n                type SLICE = flat_serialize::Slice<#rl, #ident #lifetime_args>;\n                type OWNED = #ident #owned_lifetime;\n\n                #try_ref\n\n                #fill_slice\n\n                #len\n\n                fn make_owned(&mut self) {\n                    let Self { #(#field_names,)* } = self;\n                    #(#make_owned)*\n                }\n\n                fn into_owned(self) -> Self::OWNED {\n                    let Self { #(#field_names1,)* } = self;\n                    Self::OWNED {\n                        #(#into_owned)*\n                    }\n                }\n            }\n        }\n    };\n\n    let expanded = quote! {\n        #ref_def\n    };\n\n    expanded\n}\n\nfn flat_serialize_enum(input: FlatSerializeEnum) -> TokenStream2 {\n    let alignment_check = input.alignment_check();\n    let uniqueness_check = input.uniqueness_check();\n    let trait_check = input.fn_trait_check();\n    let required_alignment = input.fn_required_alignment();\n    let max_provided_alignment = input.fn_max_provided_alignment();\n    let min_len = input.fn_min_len();\n\n    let make_owned = input.variants.iter().map(|v| {\n        let variant = &v.body.ident;\n        let fields = v.body.fields.iter().map(|f| &f.ident);\n        let make = v.body.fields.iter().map(|f| f.make_owned());\n        quote! {\n            Self::#variant { #(#fields,)* } => {\n                #(#make)*\n            },\n        }\n    });\n\n    let into_owned = input.variants.iter().map(|v| {\n        let variant = &v.body.ident;\n        let fields = v.body.fields.iter().map(|f| &f.ident);\n        let into = v.body.fields.iter().map(|f| f.into_owned());\n        quote! {\n            Self::#variant { #(#fields,)* } => Self::OWNED::#variant {\n                #(#into)*\n            },\n        }\n    });\n\n    let lifetime = input.lifetime.as_ref().map(|lifetime| quote! { #lifetime });\n    let lifetime_args = input\n        .lifetime\n        .as_ref()\n        .map(|lifetime| quote! { <#lifetime> });\n    let ref_liftime = lifetime_args.clone().unwrap_or_else(|| quote! { <'a> });\n    let rl = lifetime.clone().unwrap_or_else(|| quote! { 'a });\n    let owned_lifetime = if lifetime_args.is_some() {\n        Some(quote!( <'static> ))\n    } else {\n        None\n    };\n\n    let try_ref = input.fn_try_ref(lifetime.as_ref());\n    let fill_slice = input.fn_fill_slice();\n    let len = input.fn_len();\n    let body = input.variants(lifetime.as_ref());\n    let ident = &input.ident;\n    let attrs = &*input.attrs;\n\n    quote! {\n        #[derive(Clone)]\n        #(#attrs)*\n        #body\n\n        #alignment_check\n\n        #uniqueness_check\n\n        #trait_check\n\n        unsafe impl #ref_liftime flat_serialize::FlatSerializable #ref_liftime for #ident #lifetime_args {\n            #required_alignment\n\n            #max_provided_alignment\n\n            #min_len\n\n            // cannot be TRIVIAL_COPY since the rust enum layout is unspecified\n            const TRIVIAL_COPY: bool = false;\n            type SLICE = flat_serialize::Slice<#rl, #ident #lifetime_args>;\n            type OWNED = #ident #owned_lifetime;\n\n            #try_ref\n\n            #fill_slice\n\n            #len\n\n            fn make_owned(&mut self) {\n                match self {\n                    #(#make_owned)*\n                }\n            }\n\n            fn into_owned(self) -> Self::OWNED {\n                match self {\n                    #(#into_owned)*\n                }\n            }\n        }\n    }\n}\n\nimpl VariableLenFieldInfo {\n    fn len_from_bytes(&self) -> TokenStream2 {\n        let mut lfb = SelfReplacer(|name| syn::parse_quote! { #name.clone().unwrap() });\n        let mut len = self.len_expr.clone();\n        lfb.visit_expr_mut(&mut len);\n        quote! { #len }\n    }\n\n    fn counter_expr(&self) -> TokenStream2 {\n        let mut ce = SelfReplacer(|name| syn::parse_quote! { (*#name) });\n        let mut len = self.len_expr.clone();\n        ce.visit_expr_mut(&mut len);\n        quote! { #len }\n    }\n\n    fn err_size_expr(&self) -> TokenStream2 {\n        let mut ese = SelfReplacer(|name| {\n            syn::parse_quote! {\n                match #name { Some(#name) => #name, None => return 0usize, }\n            }\n        });\n        let mut len = self.len_expr.clone();\n        ese.visit_expr_mut(&mut len);\n        quote! { #len }\n    }\n}\n\nstruct SelfReplacer<F: FnMut(&Ident) -> syn::Expr>(F);\n\nimpl<F: FnMut(&Ident) -> syn::Expr> VisitMut for SelfReplacer<F> {\n    fn visit_expr_mut(&mut self, expr: &mut syn::Expr) {\n        if let syn::Expr::Field(field) = expr {\n            if let syn::Expr::Path(path) = &mut *field.base {\n                if path.path.segments[0].ident == \"self\" {\n                    let name = match &field.member {\n                        syn::Member::Named(name) => name,\n                        syn::Member::Unnamed(_) => panic!(\"unnamed fields not supported\"),\n                    };\n                    *expr = self.0(name)\n                }\n            }\n        } else {\n            syn::visit_mut::visit_expr_mut(self, expr)\n        }\n    }\n}\n\nstruct TryRefBody {\n    vars: TokenStream2,\n    body: TokenStream2,\n    set_fields: TokenStream2,\n    err_size: TokenStream2,\n}\n\nimpl FlatSerializeEnum {\n    fn variants(&self, lifetime: Option<&TokenStream2>) -> TokenStream2 {\n        let id = &self.ident;\n        let variants = self.variants.iter().map(|variant| {\n            let fields = variant\n                .body\n                .fields\n                .iter()\n                .map(|f| f.declaration(false, lifetime, self.per_field_attrs.iter()));\n            let ident = &variant.body.ident;\n            quote! {\n                #ident {\n                    #(#fields)*\n                },\n            }\n        });\n        let args = lifetime.map(|lifetime| quote! { <#lifetime> });\n        quote! {\n            pub enum #id #args {\n                #(#variants)*\n            }\n        }\n    }\n\n    fn uniqueness_check(&self) -> TokenStream2 {\n        let variants = self.variants.iter().map(|variant| {\n            let ident = &variant.body.ident;\n            let tag_val = &variant.tag_val;\n            quote! {\n                #ident = #tag_val,\n            }\n        });\n        quote! {\n            // uniqueness check\n            const _: () = {\n                #[allow(dead_code)]\n                enum UniquenessCheck {\n                    #(#variants)*\n                }\n            };\n        }\n    }\n\n    fn alignment_check(&self) -> TokenStream2 {\n        let tag_check = self.tag.alignment_check();\n        let variant_checks = self.variants.iter().map(|v| {\n            v.body\n                .alignment_check(quote!(current_size), quote!(min_align))\n        });\n        quote! {\n            // alignment assertions\n            #[allow(unused_assignments)]\n            const _: () = {\n                use std::mem::{align_of, size_of};\n                let mut current_size = 0;\n                let mut min_align = 8;\n                #tag_check\n                #(#variant_checks)*\n            };\n        }\n    }\n\n    fn fn_trait_check(&self) -> TokenStream2 {\n        let tag_check = self.tag.trait_check();\n        let checks = self.variants.iter().map(|v| v.body.fn_trait_check());\n        quote! {\n            const _: () = {\n                #tag_check\n                #(\n                    const _: () = {\n                        #checks\n                    };\n                )*\n            };\n        }\n    }\n\n    fn fn_required_alignment(&self) -> TokenStream2 {\n        let tag_alignment = self.tag.required_alignment();\n        let alignments = self.variants.iter().map(|v| {\n            let alignments = v.body.fields.iter().map(|f| f.required_alignment());\n            quote! {\n                let mut required_alignment = #tag_alignment;\n                #(\n                    let alignment = #alignments;\n                    if alignment > required_alignment {\n                        required_alignment = alignment;\n                    }\n                )*\n                required_alignment\n            }\n        });\n\n        quote! {\n            const REQUIRED_ALIGNMENT: usize = {\n                use std::mem::align_of;\n                let mut required_alignment: usize = #tag_alignment;\n                #(\n                    let alignment: usize = {\n                        #alignments\n                    };\n                    if alignment > required_alignment {\n                        required_alignment = alignment;\n                    }\n                )*\n                required_alignment\n            };\n        }\n    }\n\n    fn fn_max_provided_alignment(&self) -> TokenStream2 {\n        let min_align = self.tag.max_provided_alignment();\n        let min_align = quote! {\n            match #min_align {\n                Some(a) => Some(a),\n                None => Some(8),\n            }\n        };\n\n        let min_size = self.tag.min_len();\n\n        let alignments = self.variants.iter().map(|v| {\n            let alignments = v.body.fields.iter().map(|f| f.max_provided_alignment());\n            let sizes = v.body.fields.iter().map(|f| f.min_len());\n            quote! {\n                let mut min_align: Option<usize> = #min_align;\n                #(\n                    let alignment = #alignments;\n                    match (alignment, min_align) {\n                        (None, _) => (),\n                        (Some(align), None) => min_align = Some(align),\n                        (Some(align), Some(min)) if align < min =>\n                            min_align = Some(align),\n                        _ => (),\n                    }\n                )*\n                let variant_size: usize = #min_size #(+ #sizes)*;\n                let effective_alignment = match min_align {\n                    Some(align) => align,\n                    None => 8,\n                };\n\n                if variant_size % 8 == 0 && effective_alignment >= 8 {\n                    8\n                } else if variant_size % 4 == 0 && effective_alignment >= 4 {\n                    4\n                } else if variant_size % 2 == 0 && effective_alignment >= 2 {\n                    2\n                } else {\n                    1\n                }\n            }\n        });\n        quote! {\n            const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n                use std::mem::{align_of, size_of};\n                let mut min_align: usize = match #min_align {\n                    None => 8,\n                    Some(align) => align,\n                };\n                #(\n                    let variant_alignment: usize = {\n                        #alignments\n                    };\n                    if variant_alignment < min_align {\n                        min_align = variant_alignment\n                    }\n                )*\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            };\n        }\n    }\n\n    fn fn_min_len(&self) -> TokenStream2 {\n        let tag_size = self.tag.min_len();\n        let sizes = self.variants.iter().map(|v| {\n            let sizes = v.body.fields.iter().map(|f| f.min_len());\n            quote! {\n                let mut size: usize = #tag_size;\n                #(size += #sizes;)*\n                size\n            }\n        });\n        quote! {\n            const MIN_LEN: usize = {\n                use std::mem::size_of;\n                let mut size: Option<usize> = None;\n                #(\n                    let variant_size = {\n                        #sizes\n                    };\n                    size = match size {\n                        None => Some(variant_size),\n                        Some(size) if size > variant_size => Some(variant_size),\n                        Some(size) => Some(size),\n                    };\n                )*\n                match size {\n                    Some(size) => size,\n                    None => #tag_size,\n                }\n            };\n        }\n    }\n\n    fn fn_try_ref(&self, lifetime: Option<&TokenStream2>) -> TokenStream2 {\n        let break_label = syn::Lifetime::new(\"'tryref_tag\", proc_macro2::Span::call_site());\n        let try_wrap_tag = self.tag.try_wrap(&break_label);\n        let id = &self.ident;\n        let tag_ty = &self.tag.ty;\n\n        let bodies = self.variants.iter().enumerate().map(|(i, v)| {\n            let tag_val = &v.tag_val;\n\n            let variant = &v.body.ident;\n\n            let break_label =\n                syn::Lifetime::new(&format!(\"'tryref_{i}\"), proc_macro2::Span::call_site());\n\n            let TryRefBody {\n                vars,\n                body,\n                set_fields,\n                err_size,\n            } = v\n                .body\n                .fn_try_ref_body(&break_label);\n\n            quote! {\n                Some(#tag_val) => {\n                    #vars\n                    #break_label: loop {\n                        #body\n                        let _ref = #id::#variant { #set_fields };\n                        return Ok((_ref, input))\n                    }\n                    return Err(flat_serialize::WrapErr::NotEnoughBytes(std::mem::size_of::<#tag_ty>() #err_size))\n                }\n            }\n        });\n\n        let tag_ident = self.tag.ident.as_ref().unwrap();\n\n        quote! {\n            #[allow(unused_assignments, unused_variables, unreachable_code)]\n            #[inline(always)]\n            unsafe fn try_ref(mut input: & #lifetime [u8]) -> Result<(Self, & #lifetime [u8]), flat_serialize::WrapErr> {\n                let __packet_macro_read_len = 0usize;\n                let mut #tag_ident = None;\n                'tryref_tag: loop {\n                    #try_wrap_tag;\n                    match #tag_ident {\n                        #(#bodies),*\n                        _ => return Err(flat_serialize::WrapErr::InvalidTag(0)),\n                    }\n                }\n                //TODO\n                Err(flat_serialize::WrapErr::NotEnoughBytes(::std::mem::size_of::<#tag_ty>()))\n            }\n        }\n    }\n\n    fn fn_fill_slice(&self) -> TokenStream2 {\n        let tag_ty = &self.tag.ty;\n        let tag_ident = self.tag.ident.as_ref().unwrap();\n        let fill_slice_tag = self.tag.fill_slice();\n        let id = &self.ident;\n        let bodies = self.variants.iter().map(|v| {\n            let tag_val = &v.tag_val;\n            let variant = &v.body.ident;\n            let (fields, fill_slice_with) = v.body.fill_slice_body();\n            quote! {\n                #id::#variant { #fields } => {\n                    let #tag_ident: &#tag_ty = &#tag_val;\n                    #fill_slice_tag\n                    #fill_slice_with\n                }\n            }\n        });\n        quote! {\n            #[allow(unused_assignments, unused_variables)]\n            unsafe fn fill_slice<'out>(&self, input: &'out mut [std::mem::MaybeUninit<u8>])\n            -> &'out mut [std::mem::MaybeUninit<u8>] {\n                let total_len = self.num_bytes();\n                let (mut input, rem) = input.split_at_mut(total_len);\n                match self {\n                    #(#bodies),*\n                }\n                debug_assert_eq!(input.len(), 0);\n                rem\n            }\n        }\n    }\n\n    fn fn_len(&self) -> TokenStream2 {\n        let tag_ty = &self.tag.ty;\n        let tag_size = quote! { ::std::mem::size_of::<#tag_ty>() };\n        let id = &self.ident;\n        let bodies = self.variants.iter().map(|v| {\n            let variant = &v.body.ident;\n\n            let size = v.body.fields.iter().map(|f| f.size_fn());\n            let fields = v.body.fields.iter().map(|f| f.ident.as_ref().unwrap());\n            quote! {\n                #id::#variant { #(#fields),* } => {\n                    #tag_size #(+ #size)*\n                },\n            }\n        });\n        quote! {\n            #[allow(unused_assignments, unused_variables)]\n            fn num_bytes(&self) -> usize {\n                match self {\n                    #(#bodies)*\n                }\n            }\n        }\n    }\n}\n\nimpl FlatSerializeStruct {\n    fn alignment_check(&self, start: TokenStream2, min_align: TokenStream2) -> TokenStream2 {\n        let checks = self.fields.iter().map(|f| f.alignment_check());\n\n        quote! {\n            {\n                use std::mem::{align_of, size_of};\n                let mut current_size = #start;\n                let mut min_align = #min_align;\n                #(#checks)*\n            }\n        }\n    }\n\n    fn fn_trait_check(&self) -> TokenStream2 {\n        let checks = self.fields.iter().map(|f| f.trait_check());\n        quote! {\n            const _: () = {\n                #(#checks)*\n            };\n        }\n    }\n\n    fn fn_required_alignment(&self) -> TokenStream2 {\n        let alignments = self.fields.iter().map(|f| f.required_alignment());\n        quote! {\n            const REQUIRED_ALIGNMENT: usize = {\n                use std::mem::align_of;\n                let mut required_alignment = 1;\n                #(\n                    let alignment = #alignments;\n                    if alignment > required_alignment {\n                        required_alignment = alignment;\n                    }\n                )*\n                required_alignment\n            };\n        }\n    }\n\n    fn fn_max_provided_alignment(&self) -> TokenStream2 {\n        let alignments = self.fields.iter().map(|f| f.max_provided_alignment());\n        quote! {\n            const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n                use std::mem::align_of;\n                let mut min_align: Option<usize> = None;\n                #(\n                    let ty_align = #alignments;\n                    match (ty_align, min_align) {\n                        (None, _) => (),\n                        (Some(align), None) => min_align = Some(align),\n                        (Some(align), Some(min)) if align < min =>\n                            min_align = Some(align),\n                        _ => (),\n                    }\n                )*\n                match min_align {\n                    None => None,\n                    Some(min_align) => {\n                        let min_size = Self::MIN_LEN;\n                        if min_size % 8 == 0 && min_align >= 8 {\n                            Some(8)\n                        } else if min_size % 4 == 0 && min_align >= 4 {\n                            Some(4)\n                        } else if min_size % 2 == 0 && min_align >= 2 {\n                            Some(2)\n                        } else {\n                            Some(1)\n                        }\n                    },\n                }\n            };\n        }\n    }\n\n    fn fn_min_len(&self) -> TokenStream2 {\n        let sizes = self.fields.iter().map(|f| f.min_len());\n        quote! {\n            const MIN_LEN: usize = {\n                use std::mem::size_of;\n                let mut size = 0;\n                #(size += #sizes;)*\n                size\n            };\n        }\n    }\n\n    fn fn_try_ref(&self, lifetime: Option<&TokenStream2>) -> TokenStream2 {\n        let break_label = syn::Lifetime::new(\"'tryref\", proc_macro2::Span::call_site());\n        let id = &self.ident;\n        let TryRefBody {\n            vars,\n            body,\n            set_fields,\n            err_size,\n        } = self.fn_try_ref_body(&break_label);\n        quote! {\n            #[allow(unused_assignments, unused_variables)]\n            #[inline(always)]\n            unsafe fn try_ref(mut input: & #lifetime [u8])\n            -> Result<(Self, & #lifetime [u8]), flat_serialize::WrapErr> {\n                if input.len() < Self::MIN_LEN {\n                    return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN))\n                }\n                let __packet_macro_read_len = 0usize;\n                #vars\n                #break_label: loop {\n                    #body\n                    let _ref = #id { #set_fields };\n                    return Ok((_ref, input))\n                }\n                Err(flat_serialize::WrapErr::NotEnoughBytes(0 #err_size))\n            }\n        }\n    }\n\n    fn fn_try_ref_body(&self, break_label: &syn::Lifetime) -> TryRefBody {\n        let field_names = self.fields.iter().map(|f| &f.ident);\n        let ty1 = self.fields.iter().map(|f| f.local_ty());\n        let field1 = field_names.clone();\n        let field2 = field_names.clone();\n        let field_setters = self.fields.iter().map(|field| {\n            let name = &field.ident;\n            if field.is_optional() {\n                quote! { #name }\n            } else {\n                quote! { #name.unwrap() }\n            }\n        });\n\n        let vars = quote!( #(let mut #field1: #ty1 = None;)* );\n        let try_wrap_fields = self.fields.iter().map(|f| f.try_wrap(break_label));\n        let body = quote! ( #(#try_wrap_fields)* );\n\n        let set_fields = quote!( #(#field2: #field_setters),* );\n\n        let err_size = self.fields.iter().map(|f| f.err_size());\n        let err_size = quote!( #( + #err_size)* );\n        TryRefBody {\n            vars,\n            body,\n            set_fields,\n            err_size,\n        }\n    }\n\n    fn fn_fill_slice(&self) -> TokenStream2 {\n        let id = &self.ident;\n        let (fields, fill_slice_with) = self.fill_slice_body();\n        quote! {\n            #[allow(unused_assignments, unused_variables)]\n            #[inline(always)]\n            unsafe fn fill_slice<'out>(&self, input: &'out mut [std::mem::MaybeUninit<u8>]) -> &'out mut [std::mem::MaybeUninit<u8>] {\n                let total_len = self.num_bytes();\n                let (mut input, rem) = input.split_at_mut(total_len);\n                let #id { #fields } = self;\n                #fill_slice_with\n                debug_assert_eq!(input.len(), 0);\n                rem\n            }\n        }\n    }\n    fn fill_slice_body(&self) -> (TokenStream2, TokenStream2) {\n        //FIXME assert multiple values of counters are equal...\n        let fill_slice_with = self.fields.iter().map(|f| f.fill_slice());\n        let fill_slice_with = quote!( #(#fill_slice_with);* );\n\n        let field = self.fields.iter().map(|f| f.ident.as_ref().unwrap());\n        let fields = quote!( #(#field),* );\n        (fields, fill_slice_with)\n    }\n\n    fn fn_len(&self) -> TokenStream2 {\n        let size = self.fields.iter().map(|f| f.size_fn());\n        let field = self.fields.iter().map(|f| f.ident.as_ref().unwrap());\n        let id = &self.ident;\n\n        quote! {\n            #[allow(unused_assignments, unused_variables)]\n            #[inline(always)]\n            fn num_bytes(&self) -> usize {\n                let #id { #(#field),* } = self;\n                0usize #(+ #size)*\n            }\n        }\n    }\n}\n\nimpl FlatSerializeField {\n    fn alignment_check(&self) -> TokenStream2 {\n        let current_size = quote!(current_size);\n        let min_align = quote!(min_align);\n        match &self.length_info {\n            None => {\n                let ty = self.ty_without_lifetime();\n                quote_spanned! {self.ty.span()=>\n                    if (#current_size) % <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT != 0 {\n                        panic!(\"unaligned field: the current size of the data is not a multiple of this type's alignment\")\n                    }\n                    if <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > #min_align {\n                        panic!(\"unaligned field: an earlier variable-length field could mis-align this field\")\n                    }\n                    #current_size += <#ty as flat_serialize::FlatSerializable>::MIN_LEN;\n                    #min_align = match <#ty as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                        Some(align) if align < #min_align => align,\n                        _ => #min_align,\n                    };\n                }\n            }\n            Some(info) => {\n                let ty = info.ty_without_lifetime();\n                quote_spanned! {self.ty.span()=>\n                    if (#current_size) % <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT != 0 {\n                        panic!(\"unaligned field: the current size of the data is not a multiple of this type's alignment\")\n                    }\n                    if <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT > #min_align {\n                        panic!(\"unaligned field: an earlier variable-length field could mis-align this field\")\n                    }\n                    if <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT < #min_align {\n                        #min_align = <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n                    }\n                    #min_align = match <#ty as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT {\n                        Some(align) if align < #min_align => align,\n                        _ => #min_align,\n                    };\n                }\n            }\n        }\n    }\n\n    fn trait_check(&self) -> TokenStream2 {\n        let (ty, needs_lifetime) = match (&self.ty_without_lifetime, &self.length_info) {\n            (\n                _,\n                Some(VariableLenFieldInfo {\n                    ty_without_lifetime: Some(ty),\n                    ..\n                }),\n            ) => (ty.clone(), true),\n            (_, Some(VariableLenFieldInfo { ty, .. })) => (quote! { #ty }, false),\n            (Some(ty), _) => (ty.clone(), true),\n            _ => {\n                let ty = &self.ty;\n                (quote! { #ty }, false)\n            }\n        };\n        let lifetime = needs_lifetime.then(|| quote! { <'static> });\n        let name = self.ident.as_ref().unwrap();\n        // based on static_assertions\n        // TODO add ConstLen assertion if type is in var-len position?\n        quote_spanned! {self.ty.span()=>\n            fn #name<'test, T: flat_serialize::FlatSerializable<'test>>() {}\n            let _ = #name::<#ty #lifetime>;\n        }\n    }\n\n    fn required_alignment(&self) -> TokenStream2 {\n        let ty = match &self.length_info {\n            None => self.ty_without_lifetime(),\n            Some(info) => info.ty_without_lifetime(),\n        };\n        quote_spanned! {self.ty.span()=>\n            <#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT\n        }\n    }\n\n    fn max_provided_alignment(&self) -> TokenStream2 {\n        match &self.length_info {\n            None => {\n                let ty = self.ty_without_lifetime();\n                quote_spanned! {self.ty.span()=>\n                    <#ty as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT\n                }\n            }\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: true, ..\n                },\n            ) => {\n                let ty = info.ty_without_lifetime();\n                // fields after an optional field cannot be aligned to more than\n                // the field is in the event the field is present, so if the\n                // field does not provide a max alignment (i.e. it's fixed-len)\n                // use that to determine what the max alignment is.\n                quote_spanned! {self.ty.span()=>\n                    {\n                        let ty_provied = <#ty as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n                        match ty_provied {\n                            Some(align) => Some(align),\n                            None => Some(<#ty as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT),\n                        }\n                    }\n                }\n            }\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: false, ..\n                },\n            ) => {\n                let ty = info.ty_without_lifetime();\n                // for variable length slices we only need to check the required\n                // alignment, not the max-provided: TRIVIAL_COPY types won't\n                // have a max-provided alignment, while other ones will be\n                // padded out to their natural alignment.\n                quote_spanned! {self.ty.span()=>\n                    {\n                        Some(<#ty as  flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT)\n                    }\n                }\n            }\n        }\n    }\n\n    fn min_len(&self) -> TokenStream2 {\n        match &self.length_info {\n            None => {\n                let ty = self.ty_without_lifetime();\n                quote_spanned! {self.ty.span()=>\n                    <#ty as flat_serialize::FlatSerializable>::MIN_LEN\n                }\n            }\n            Some(..) => quote_spanned! {self.ty.span()=>\n                0\n            },\n        }\n    }\n\n    fn try_wrap(&self, break_label: &syn::Lifetime) -> TokenStream2 {\n        let ident = self.ident.as_ref().unwrap();\n        match &self.length_info {\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: false, ..\n                },\n            ) => {\n                let count = info.len_from_bytes();\n                quote! {\n                    {\n                        let count = (#count) as usize;\n                        let (field, rem) = match <_ as flat_serialize::VariableLen <'_\n                        >>::try_ref(input, count) {\n                            Ok((f, b)) => (f, b),\n                            Err(flat_serialize::WrapErr::InvalidTag(offset)) =>\n                                return Err(flat_serialize::WrapErr::InvalidTag(__packet_macro_read_len + offset)),\n                            Err(..) => break #break_label\n                        };\n                        input = rem;\n                        #ident = Some(field);\n                    }\n                }\n            }\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: true, ..\n                },\n            ) => {\n                let is_present = info.len_from_bytes();\n                let ty = info.ty_without_lifetime();\n                quote! {\n                    if #is_present {\n                        let (field, rem) = match <#ty>::try_ref(input) {\n                            Ok((f, b)) => (f, b),\n                            Err(flat_serialize::WrapErr::InvalidTag(offset)) =>\n                                return Err(flat_serialize::WrapErr::InvalidTag(__packet_macro_read_len + offset)),\n                            Err(..) => break #break_label\n\n                        };\n                        input = rem;\n                        #ident = Some(field);\n                    }\n                }\n            }\n            None => {\n                let ty = self.ty_without_lifetime();\n                quote! {\n                    {\n                        let (field, rem) = match <#ty>::try_ref(input) {\n                            Ok((f, b)) => (f, b),\n                            Err(flat_serialize::WrapErr::InvalidTag(offset)) =>\n                                return Err(flat_serialize::WrapErr::InvalidTag(__packet_macro_read_len + offset)),\n                            Err(..) => break #break_label\n\n                        };\n                        input = rem;\n                        #ident = Some(field);\n                    }\n                }\n            }\n        }\n    }\n\n    fn fill_slice(&self) -> TokenStream2 {\n        let ident = self.ident.as_ref().unwrap();\n        match &self.length_info {\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: false, ..\n                },\n            ) => {\n                let count = info.counter_expr();\n                // TODO this may not elide all bounds checks\n                quote! {\n                    unsafe {\n                        let count = (#count) as usize;\n                        input = <_ as flat_serialize::VariableLen<'_>>::fill_slice(#ident, count, input);\n                    }\n                }\n            }\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: true, ..\n                },\n            ) => {\n                let is_present = info.counter_expr();\n                let ty = &info.ty;\n                quote! {\n                    unsafe {\n                        if #is_present {\n                            let #ident: &#ty = #ident.as_ref().unwrap();\n                            input = #ident.fill_slice(input);\n                        }\n                    }\n                }\n            }\n            None => {\n                quote! {\n                    unsafe {\n                        input = #ident.fill_slice(input);\n                    }\n                }\n            }\n        }\n    }\n\n    fn err_size(&self) -> TokenStream2 {\n        match &self.length_info {\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: false, ..\n                },\n            ) => {\n                let count = info.err_size_expr();\n                let ty = info.ty_without_lifetime();\n                quote! {\n                    (|| <#ty>::MIN_LEN * (#count) as usize)()\n                }\n            }\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: true, ..\n                },\n            ) => {\n                let is_present = info.err_size_expr();\n                let ty = info.ty_without_lifetime();\n                quote! {\n                    (|| if #is_present { <#ty>::MIN_LEN } else { 0 })()\n                }\n            }\n            None => {\n                let ty = &self.ty_without_lifetime();\n                quote! { <#ty>::MIN_LEN }\n            }\n        }\n    }\n\n    fn exposed_ty(&self, lifetime: Option<&TokenStream2>) -> TokenStream2 {\n        match &self.length_info {\n            None => {\n                let nominal_ty = &self.ty;\n                quote_spanned! {self.field.span()=>\n                    #nominal_ty\n                }\n            }\n            Some(VariableLenFieldInfo {\n                is_optional: false,\n                ty,\n                ..\n            }) => quote_spanned! {self.field.span()=>\n                <#ty as flat_serialize::FlatSerializable<#lifetime>>::SLICE\n            },\n            Some(VariableLenFieldInfo {\n                is_optional: true,\n                ty,\n                ..\n            }) => {\n                quote_spanned! {self.field.span()=>\n                    Option<#ty>\n                }\n            }\n        }\n    }\n\n    fn local_ty(&self) -> TokenStream2 {\n        match &self.length_info {\n            None => {\n                let ty = &self.ty;\n                quote! { Option<#ty> }\n            }\n            Some(VariableLenFieldInfo {\n                is_optional: false,\n                ty,\n                ..\n            }) => {\n                quote! { Option<<#ty as flat_serialize::FlatSerializable<'_>>::SLICE> }\n            }\n            Some(VariableLenFieldInfo {\n                is_optional: true,\n                ty,\n                ..\n            }) => {\n                quote! { Option<#ty> }\n            }\n        }\n    }\n\n    fn size_fn(&self) -> TokenStream2 {\n        let ident = self.ident.as_ref().unwrap();\n        match &self.length_info {\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: false, ..\n                },\n            ) => {\n                let count = info.counter_expr();\n                quote! {\n                    (<_ as flat_serialize::VariableLen<'_>>::num_bytes(#ident, (#count) as usize))\n                }\n            }\n            Some(\n                info @ VariableLenFieldInfo {\n                    is_optional: true, ..\n                },\n            ) => {\n                let ty = self.ty_without_lifetime();\n                let is_present = info.counter_expr();\n                quote! {\n                    (if #is_present {\n                        <#ty as flat_serialize::FlatSerializable>::num_bytes(#ident.as_ref().unwrap())\n                    } else {\n                        0\n                    })\n                }\n            }\n            None => {\n                let nominal_ty = self.ty_without_lifetime();\n                quote!( <#nominal_ty as flat_serialize::FlatSerializable>::num_bytes(#ident) )\n            }\n        }\n    }\n\n    fn make_owned(&self) -> TokenStream2 {\n        let ident = self.ident.as_ref().unwrap();\n        match &self.length_info {\n            Some(VariableLenFieldInfo {\n                is_optional: false, ..\n            }) => {\n                quote! { flat_serialize::Slice::make_owned(#ident); }\n            }\n            Some(VariableLenFieldInfo {\n                is_optional: true, ..\n            }) => {\n                let ty = self.ty_without_lifetime();\n                quote! {\n                    #ident.as_mut().map(|v| <#ty as flat_serialize::FlatSerializable>::make_owned(v));\n                }\n            }\n            None => {\n                let nominal_ty = self.ty_without_lifetime();\n                quote!( <#nominal_ty as flat_serialize::FlatSerializable>::make_owned(#ident); )\n            }\n        }\n    }\n\n    #[allow(clippy::wrong_self_convention)]\n    fn into_owned(&self) -> TokenStream2 {\n        let ident = self.ident.as_ref().unwrap();\n        match &self.length_info {\n            Some(VariableLenFieldInfo {\n                is_optional: false, ..\n            }) => {\n                quote! { #ident: flat_serialize::Slice::into_owned(#ident), }\n            }\n            Some(VariableLenFieldInfo {\n                is_optional: true, ..\n            }) => {\n                let ty = self.ty_without_lifetime();\n                quote! {\n                    #ident: #ident.map(|v| <#ty as flat_serialize::FlatSerializable>::into_owned(v)),\n                }\n            }\n            None => {\n                let nominal_ty = self.ty_without_lifetime();\n                quote!( #ident: <#nominal_ty as flat_serialize::FlatSerializable>::into_owned(#ident), )\n            }\n        }\n    }\n\n    fn declaration<'a, 'b: 'a>(\n        &'b self,\n        is_pub: bool,\n        lifetime: Option<&TokenStream2>,\n        pf_attrs: impl Iterator<Item = &'a PerFieldsAttr> + 'a,\n    ) -> TokenStream2 {\n        let name = self.ident.as_ref().unwrap();\n        let attrs = self.attrs.iter();\n        let pub_marker = is_pub.then(|| quote! { pub });\n        let ty = self.exposed_ty(lifetime);\n        let per_field_attrs = self.per_field_attrs(pf_attrs);\n        quote! { #(#per_field_attrs)* #(#attrs)* #pub_marker #name: #ty, }\n    }\n\n    fn per_field_attrs<'a, 'b: 'a>(\n        &'b self,\n        attrs: impl Iterator<Item = &'a PerFieldsAttr> + 'a,\n    ) -> impl Iterator<Item = TokenStream2> + 'a {\n        attrs.map(move |attr| match &self.length_info {\n            None => {\n                let attr = &attr.fixed;\n                quote! { #attr }\n            }\n            Some(_) => match &attr.variable {\n                Some(attr) => quote! { #attr },\n                None => quote! {},\n            },\n        })\n    }\n\n    fn ty_without_lifetime(&self) -> TokenStream2 {\n        match &self.ty_without_lifetime {\n            None => {\n                let ty = &self.ty;\n                quote! { #ty }\n            }\n            Some(ty) => ty.clone(),\n        }\n    }\n\n    fn is_optional(&self) -> bool {\n        matches!(\n            self.length_info,\n            Some(VariableLenFieldInfo {\n                is_optional: true,\n                ..\n            })\n        )\n    }\n}\n\nimpl VariableLenFieldInfo {\n    fn ty_without_lifetime(&self) -> TokenStream2 {\n        match &self.ty_without_lifetime {\n            None => {\n                let ty = &self.ty;\n                quote! { #ty }\n            }\n            Some(ty) => ty.clone(),\n        }\n    }\n}\n\n#[proc_macro_derive(FlatSerializable)]\npub fn flat_serializable_derive(input: TokenStream) -> TokenStream {\n    let input: syn::DeriveInput = syn::parse(input).unwrap();\n    let name = input.ident;\n\n    let s = match input.data {\n        syn::Data::Enum(e) => {\n            let repr: Vec<_> = input\n                .attrs\n                .iter()\n                .flat_map(|attr| {\n                    let meta = match attr.parse_meta() {\n                        Ok(meta) => meta,\n                        _ => return None,\n                    };\n                    let has_repr = meta.path().get_ident().is_some_and(|id| id == \"repr\");\n                    if !has_repr {\n                        return None;\n                    }\n                    attr.parse_args().ok().and_then(|ident: Ident| {\n                        if ident == \"u8\" || ident == \"u16\" || ident == \"u32\" || ident == \"u64\" {\n                            return Some(ident);\n                        }\n                        None\n                    })\n                })\n                .collect();\n            if repr.len() != 1 {\n                return quote_spanned! {e.enum_token.span()=>\n                    compile_error!{\"FlatSerializable only allowed on #[repr(u..)] enums without variants\"}\n                }.into();\n            }\n            let all_unit = e\n                .variants\n                .iter()\n                .all(|variant| matches!(variant.fields, syn::Fields::Unit));\n            if !all_unit {\n                return quote_spanned! {e.enum_token.span()=>\n                    compile_error!{\"FlatSerializable only allowed on until enums\"}\n                }\n                .into();\n            }\n\n            let variant = e.variants.iter().map(|v| &v.ident);\n            let variant2 = variant.clone();\n            let const_name = variant.clone();\n            let repr = &repr[0];\n\n            let out = quote! {\n                unsafe impl<'i> flat_serialize::FlatSerializable<'i> for #name {\n                    const MIN_LEN: usize = std::mem::size_of::<Self>();\n                    const REQUIRED_ALIGNMENT: usize = std::mem::align_of::<Self>();\n                    const MAX_PROVIDED_ALIGNMENT: Option<usize> = None;\n                    const TRIVIAL_COPY: bool = true;\n                    type SLICE = flat_serialize::Slice<'i, #name>;\n                    type OWNED = Self;\n\n                    #[inline(always)]\n                    #[allow(non_upper_case_globals)]\n                    unsafe fn try_ref(input: &'i [u8])\n                    -> Result<(Self, &'i [u8]), flat_serialize::WrapErr> {\n                        let size = std::mem::size_of::<Self>();\n                        if input.len() < size {\n                            return Err(flat_serialize::WrapErr::NotEnoughBytes(size))\n                        }\n                        let (field, rem) = input.split_at(size);\n                        let field = field.as_ptr().cast::<#repr>();\n                        #(\n                            const #const_name: #repr = #name::#variant2 as #repr;\n                        )*\n                        let field = field.read_unaligned();\n                        let field = match field {\n                            #(#variant => #name::#variant,)*\n                            _ => return Err(flat_serialize::WrapErr::InvalidTag(0)),\n                        };\n                        Ok((field, rem))\n                    }\n\n                    #[inline(always)]\n                    unsafe fn fill_slice<'out>(&self, input: &'out mut [std::mem::MaybeUninit<u8>])\n                    -> &'out mut [std::mem::MaybeUninit<u8>] {\n                        let size = std::mem::size_of::<Self>();\n                        let (input, rem) = input.split_at_mut(size);\n                        let bytes = (self as *const Self).cast::<std::mem::MaybeUninit<u8>>();\n                        let bytes = std::slice::from_raw_parts(bytes, size);\n                        input.copy_from_slice(bytes);\n                        rem\n                    }\n\n                    #[inline(always)]\n                    fn num_bytes(&self) -> usize {\n                        std::mem::size_of::<Self>()\n                    }\n\n                    #[inline(always)]\n                    fn make_owned(&mut self) {\n                        // nop\n                    }\n\n                    #[inline(always)]\n                    fn into_owned(self) -> Self::OWNED {\n                        self\n                    }\n                }\n            };\n            return out.into();\n        }\n        syn::Data::Union(u) => {\n            return quote_spanned! {u.union_token.span()=>\n                compile_error!(\"FlatSerializable not allowed on unions\")\n            }\n            .into()\n        }\n        syn::Data::Struct(s) => s,\n    };\n\n    let num_reprs = input\n        .attrs\n        .iter()\n        .flat_map(|attr| {\n            let meta = match attr.parse_meta() {\n                Ok(meta) => meta,\n                _ => return None,\n            };\n            let has_repr = meta.path().get_ident().is_some_and(|id| id == \"repr\");\n            if !has_repr {\n                return None;\n            }\n            attr.parse_args().ok().and_then(|ident: Ident| {\n                if ident == \"C\" {\n                    return Some(ident);\n                }\n                None\n            })\n        })\n        .count();\n    if num_reprs != 1 {\n        return quote_spanned! {s.struct_token.span()=>\n            compile_error!{\"FlatSerializable only allowed on #[repr(C)] structs\"}\n        }\n        .into();\n    }\n\n    let s = FlatSerializeStruct {\n        per_field_attrs: Default::default(),\n        attrs: Default::default(),\n        ident: name,\n        lifetime: None,\n        fields: s\n            .fields\n            .into_iter()\n            .map(|f| FlatSerializeField {\n                field: f,\n                ty_without_lifetime: None,\n                length_info: None,\n            })\n            .collect(),\n    };\n\n    let ident = &s.ident;\n    let alignment_check = s.alignment_check(quote!(0), quote!(8));\n    let trait_check = s.fn_trait_check();\n    let required_alignment = s.fn_required_alignment();\n    let max_provided_alignment = s.fn_max_provided_alignment();\n    let min_len = s.fn_min_len();\n\n    let try_ref = s.fn_try_ref(None);\n    let fill_slice = s.fn_fill_slice();\n    let len = s.fn_len();\n\n    // FIXME add check that all values are TRIVIAL_COPY\n    let out = quote! {\n\n        // alignment assertions\n        #[allow(unused_assignments)]\n        const _: () = #alignment_check;\n\n        #trait_check\n\n        unsafe impl<'a> flat_serialize::FlatSerializable<'a> for #ident {\n            #required_alignment\n\n            #max_provided_alignment\n\n            #min_len\n\n            const TRIVIAL_COPY: bool = true;\n            type SLICE = flat_serialize::Slice<'a, #ident>;\n            type OWNED = Self;\n\n            #try_ref\n\n            #fill_slice\n\n            #len\n\n            #[inline(always)]\n            fn make_owned(&mut self) {\n                // nop\n            }\n\n            #[inline(always)]\n            fn into_owned(self) -> Self::OWNED {\n                self\n            }\n        }\n    };\n    out.into()\n}\n"
  },
  {
    "path": "crates/flat_serialize/flat_serialize_macro/src/parser.rs",
    "content": "use std::{collections::HashSet, ops::Deref};\n\nuse proc_macro2::TokenStream as TokenStream2;\n\nuse syn::{\n    braced,\n    parse::{Parse, ParseStream},\n    spanned::Spanned,\n    token,\n    visit::Visit,\n    Attribute, Expr, Field, Ident, Result, Token, Type,\n};\n\nuse crate::{\n    FlatSerialize, FlatSerializeEnum, FlatSerializeField, FlatSerializeStruct,\n    FlatSerializeVariant, PerFieldsAttr, VariableLenFieldInfo,\n};\n\nuse quote::{quote, quote_spanned};\n\nconst LIBRARY_MARKER: &str = \"flat_serialize\";\n\nfn flat_serialize_attr_path(att_name: &str) -> syn::Path {\n    let crate_name = quote::format_ident!(\"{}\", LIBRARY_MARKER);\n    let att_name = quote::format_ident!(\"{}\", att_name);\n    syn::parse_quote! { #crate_name :: #att_name }\n}\n\nimpl Parse for FlatSerialize {\n    fn parse(input: ParseStream) -> Result<Self> {\n        let attrs = input.call(Attribute::parse_outer)?;\n        let field_attr_path = flat_serialize_attr_path(\"field_attr\");\n        let (per_field_attrs, attrs): (Vec<_>, _) = attrs\n            .into_iter()\n            .partition(|attr| attr.path == field_attr_path);\n        let per_field_attrs: Result<_> = per_field_attrs\n            .into_iter()\n            .map(|a| a.parse_args_with(PerFieldsAttr::parse))\n            .collect();\n        let per_field_attrs = per_field_attrs?;\n        let lookahead = input.lookahead1();\n        //TODO Visibility\n        if lookahead.peek(Token![struct]) {\n            input.parse().map(|mut s: FlatSerializeStruct| {\n                s.per_field_attrs = per_field_attrs;\n                s.attrs = attrs;\n                FlatSerialize::Struct(s)\n            })\n        } else if lookahead.peek(Token![enum]) {\n            input.parse().map(|mut e: FlatSerializeEnum| {\n                e.per_field_attrs = per_field_attrs;\n                e.attrs = attrs;\n                FlatSerialize::Enum(e)\n            })\n        } else {\n            Err(lookahead.error())\n        }\n    }\n}\n\nimpl Parse for FlatSerializeStruct {\n    fn parse(input: ParseStream) -> Result<Self> {\n        let content;\n        let _struct_token: Token![struct] = input.parse()?;\n        let ident = input.parse()?;\n        let mut lifetime = None;\n        if input.peek(Token![<]) {\n            let _: Token![<] = input.parse()?;\n            lifetime = Some(input.parse()?);\n            let _: Token![>] = input.parse()?;\n        }\n        let _brace_token: token::Brace = braced!(content in input);\n        let mut fields = content.parse_terminated(FlatSerializeField::parse)?;\n        validate_self_fields(fields.iter_mut());\n        Ok(Self {\n            per_field_attrs: vec![],\n            attrs: vec![],\n            ident,\n            lifetime,\n            fields,\n        })\n    }\n}\n\nimpl Parse for FlatSerializeEnum {\n    fn parse(input: ParseStream) -> Result<Self> {\n        let content;\n        let _enum_token: Token![enum] = input.parse()?;\n        let ident = input.parse()?;\n        let mut lifetime = None;\n        if input.peek(Token![<]) {\n            let _: Token![<] = input.parse()?;\n            lifetime = Some(input.parse()?);\n            let _: Token![>] = input.parse()?;\n        }\n        let _brace_token: token::Brace = braced!(content in input);\n        let tag = Field::parse_named(&content)?;\n        let _comma_token: Token![,] = content.parse()?;\n        let variants = content.parse_terminated(FlatSerializeVariant::parse)?;\n        Ok(Self {\n            per_field_attrs: vec![],\n            attrs: vec![],\n            ident,\n            lifetime,\n            tag: FlatSerializeField {\n                field: tag,\n                // TODO can we allow these?\n                ty_without_lifetime: None,\n                length_info: None,\n            },\n            variants,\n        })\n    }\n}\n\nimpl Parse for FlatSerializeVariant {\n    fn parse(input: ParseStream) -> Result<Self> {\n        let content;\n        let ident = input.parse()?;\n        let _colon_token: Token![:] = input.parse()?;\n        let tag_val = input.parse()?;\n        let _brace_token: token::Brace = braced!(content in input);\n        let mut fields = content.parse_terminated(FlatSerializeField::parse)?;\n        validate_self_fields(fields.iter_mut());\n        Ok(Self {\n            tag_val,\n            body: FlatSerializeStruct {\n                per_field_attrs: vec![],\n                attrs: vec![],\n                ident,\n                lifetime: None,\n                fields,\n            },\n        })\n    }\n}\n\nimpl Parse for FlatSerializeField {\n    fn parse(input: ParseStream) -> Result<Self> {\n        let mut field = Field::parse_named(input)?;\n        // TODO switch to `drain_filter()` once stable\n        let path = flat_serialize_attr_path(\"flatten\");\n        let mut use_trait = false;\n        field.attrs.retain(|attr| {\n            let is_flatten = attr.path == path;\n            if is_flatten {\n                use_trait = true;\n                return false;\n            }\n            true\n        });\n        let mut length_info = None;\n        if input.peek(Token![if]) {\n            let _: Token![if] = input.parse()?;\n            let expr = input.parse()?;\n            length_info = Some(VariableLenFieldInfo {\n                ty: field.ty.clone(),\n                ty_without_lifetime: None,\n                len_expr: expr,\n                is_optional: true,\n            });\n        } else if let syn::Type::Array(array) = &field.ty {\n            let has_self = has_self_field(&array.len);\n            if has_self {\n                // let self_fields_are_valid = validate_self_field(&array.len, &seen_fields);\n                length_info = Some(VariableLenFieldInfo {\n                    ty: (*array.elem).clone(),\n                    ty_without_lifetime: None,\n                    len_expr: array.len.clone(),\n                    is_optional: false,\n                });\n            }\n        }\n\n        let mut ty_without_lifetime = None;\n        if has_lifetime(&field.ty) {\n            match &mut length_info {\n                None => ty_without_lifetime = Some(as_turbofish(&field.ty)),\n                Some(info) => {\n                    info.ty_without_lifetime = Some(as_turbofish(&info.ty));\n                }\n            }\n        }\n        Ok(Self {\n            field,\n            ty_without_lifetime,\n            length_info,\n        })\n    }\n}\n\n// TODO should we leave this in?\nimpl Deref for FlatSerializeField {\n    type Target = Field;\n\n    fn deref(&self) -> &Self::Target {\n        &self.field\n    }\n}\n\nimpl Parse for PerFieldsAttr {\n    fn parse(input: ParseStream) -> Result<Self> {\n        let fixed: syn::MetaNameValue = input.parse()?;\n        let mut variable: Option<syn::MetaNameValue> = None;\n        if !input.is_empty() {\n            let _comma_token: Token![,] = input.parse()?;\n            if !input.is_empty() {\n                variable = Some(input.parse()?)\n            }\n            if !input.is_empty() {\n                let _comma_token: Token![,] = input.parse()?;\n            }\n        }\n\n        if !fixed.path.is_ident(\"fixed\") {\n            return Err(syn::Error::new(fixed.path.span(), \"expected `fixed`\"));\n        }\n        if !variable\n            .as_ref()\n            .map(|v| v.path.is_ident(\"variable\"))\n            .unwrap_or(true)\n        {\n            return Err(syn::Error::new(\n                variable.unwrap().path.span(),\n                \"expected `variable`\",\n            ));\n        }\n        let fixed = match &fixed.lit {\n            syn::Lit::Str(fixed) => {\n                let mut fixed_attrs = fixed.parse_with(Attribute::parse_outer)?;\n                if fixed_attrs.len() != 1 {\n                    return Err(syn::Error::new(\n                        fixed.span(),\n                        \"must contain exactly one attribute\",\n                    ));\n                }\n                fixed_attrs.pop().unwrap()\n            }\n\n            _ => {\n                return Err(syn::Error::new(\n                    fixed.lit.span(),\n                    \"must contain exactly one attribute\",\n                ))\n            }\n        };\n\n        let variable = match variable {\n            None => None,\n            Some(variable) => match &variable.lit {\n                syn::Lit::Str(variable) => {\n                    let mut variable_attrs = variable.parse_with(Attribute::parse_outer)?;\n                    if variable_attrs.len() != 1 {\n                        return Err(syn::Error::new(\n                            variable.span(),\n                            \"must contain exactly one attribute\",\n                        ));\n                    }\n                    Some(variable_attrs.pop().unwrap())\n                }\n\n                _ => {\n                    return Err(syn::Error::new(\n                        variable.lit.span(),\n                        \"must contain exactly one attribute\",\n                    ))\n                }\n            },\n        };\n\n        Ok(Self { fixed, variable })\n    }\n}\n\nfn has_self_field(expr: &Expr) -> bool {\n    let mut has_self = FindSelf(false);\n    has_self.visit_expr(expr);\n    has_self.0\n}\n\nstruct FindSelf(bool);\n\nimpl<'ast> Visit<'ast> for FindSelf {\n    fn visit_path_segment(&mut self, i: &'ast syn::PathSegment) {\n        if self.0 {\n            return;\n        }\n        self.0 |= i.ident == \"self\"\n    }\n}\n\n/// validate that all references to a field in the struct (e.g. `len` in\n/// `[u8; self.len + 1]`) contained in expression refers to already defined\n/// fields. Otherwise output a \"attempting to use field before definition\"\n/// compile error. This is used to ensure that wse don't generate structs that\n/// are impossible to deserialize because fields are in ambiguous positions such\n/// as\n/// ```skip\n/// struct {\n///     variable: [u8; self.len],\n///     len: u32,\n/// }\n/// ```\n/// where the position of `len` depends on the value of `len`.\nfn validate_self_fields<'a>(fields: impl Iterator<Item = &'a mut FlatSerializeField>) {\n    let mut seen_fields = HashSet::new();\n\n    for f in fields {\n        if let Some(length_info) = &mut f.length_info {\n            if let Err(error) = validate_self_field(&length_info.len_expr, &seen_fields) {\n                length_info.len_expr = syn::parse2(error).unwrap()\n            }\n        }\n        seen_fields.insert(f.ident.as_ref().unwrap());\n    }\n}\n\nfn validate_self_field(\n    expr: &Expr,\n    seen_fields: &HashSet<&Ident>,\n) -> std::result::Result<(), TokenStream2> {\n    let mut validate_fields = ValidateLenFields(None, seen_fields);\n    validate_fields.visit_expr(expr);\n    match validate_fields.0 {\n        Some(error) => Err(error),\n        None => Ok(()),\n    }\n}\n\nstruct ValidateLenFields<'a, 'b>(Option<TokenStream2>, &'b HashSet<&'a Ident>);\n\nimpl<'ast> Visit<'ast> for ValidateLenFields<'_, '_> {\n    fn visit_expr(&mut self, expr: &'ast syn::Expr) {\n        if self.0.is_some() {\n            return;\n        }\n        match expr {\n            syn::Expr::Field(field) => {\n                if let syn::Expr::Path(path) = &*field.base {\n                    if path.path.segments[0].ident == \"self\" {\n                        let name = match &field.member {\n                            syn::Member::Named(name) => name.clone(),\n                            syn::Member::Unnamed(_) => panic!(\"unnamed fields not supported\"),\n                        };\n                        if !self.1.contains(&name) {\n                            self.0 = Some(quote_spanned! {name.span()=>\n                                compile_error!(\"attempting to use field before definition\")\n                            })\n                        }\n                    }\n                }\n            }\n            _ => syn::visit::visit_expr(self, expr),\n        }\n    }\n}\n\npub fn as_turbofish(ty: &Type) -> TokenStream2 {\n    let path = match &ty {\n        Type::Path(path) => path,\n        _ => {\n            return quote_spanned! {ty.span()=>\n                compile_error!(\"can only flatten path-based types\")\n            }\n        }\n    };\n    if path.qself.is_some() {\n        return quote_spanned! {ty.span()=>\n            compile_error!(\"cannot use `<Foo as Bar>` in flatten\")\n        };\n    }\n    let path = &path.path;\n    let leading_colon = &path.leading_colon;\n    let mut output = quote! {};\n    let mut error = None;\n    for segment in &path.segments {\n        match &segment.arguments {\n            syn::PathArguments::Parenthesized(args) => {\n                error = Some(quote_spanned! {args.span()=>\n                    compile_error!(\"cannot use `()` in flatten\")\n                });\n            }\n            syn::PathArguments::None => {\n                if output.is_empty() {\n                    output = quote! { #leading_colon #segment };\n                } else {\n                    output = quote! { #output::#segment};\n                }\n            }\n            syn::PathArguments::AngleBracketed(_) => {\n                let ident = &segment.ident;\n                if output.is_empty() {\n                    // TODO leave in args?\n                    // output = quote!{ #leading_colon #ident::#args };\n                    output = quote! { #leading_colon #ident };\n                } else {\n                    // TODO leave in args?\n                    // output = quote!{ #output::#ident::#args };\n                    output = quote! { #output::#ident };\n                }\n            }\n        }\n    }\n    if let Some(error) = error {\n        return error;\n    }\n\n    output\n}\n\npub fn has_lifetime(ty: &Type) -> bool {\n    struct Visitor(bool);\n    impl<'ast> Visit<'ast> for Visitor {\n        fn visit_lifetime(&mut self, _: &'ast syn::Lifetime) {\n            self.0 = true\n        }\n    }\n    let mut visit = Visitor(false);\n    syn::visit::visit_type(&mut visit, ty);\n    visit.0\n}\n"
  },
  {
    "path": "crates/hyperloglogplusplus/Cargo.toml",
    "content": "[package]\nname = \"hyperloglogplusplus\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nserde = { version = \"1.0\", features = [\"derive\"] }\nencodings = { path=\"../encodings\" }\n\n[dev-dependencies]\nfnv = \"1.0.3\"\nquickcheck = \"1\"\nquickcheck_macros = \"1\"\n\n[features]\ndefault = []\nflaky_tests = []"
  },
  {
    "path": "crates/hyperloglogplusplus/src/dense.rs",
    "content": "use crate::hyperloglog_data::{\n    BIAS_DATA_OFFSET, BIAS_DATA_VEC, RAW_ESTIMATE_DATA_OFFSET, RAW_ESTIMATE_DATA_VEC,\n    THRESHOLD_DATA_OFFSET, THRESHOLD_DATA_VEC,\n};\n\nuse crate::{registers::Registers, Extractable};\n\n#[derive(Clone, serde::Serialize, serde::Deserialize, PartialEq, Eq)]\npub struct Storage<'s> {\n    pub registers: Registers<'s>,\n    // TODO can be derived from block.len()\n    index_shift: u8,\n    pub precision: u8,\n    hash_mask: u64,\n}\n\nimpl<'s> Storage<'s> {\n    pub fn new(precision: u8) -> Self {\n        // TODO what is max precision\n        assert!(\n            (4..=18).contains(&precision),\n            \"invalid value for precision: {precision}; must be within [4, 18]\",\n        );\n        let non_index_bits = 64 - precision;\n        Self {\n            registers: Registers::new(precision),\n            index_shift: non_index_bits,\n            precision,\n            hash_mask: (1 << non_index_bits) - 1,\n        }\n    }\n\n    pub fn from_parts(registers: &'s [u8], precision: u8) -> Self {\n        let non_index_bits = 64 - precision;\n        Self {\n            registers: Registers::from_raw(registers),\n            index_shift: non_index_bits,\n            precision,\n            hash_mask: (1 << non_index_bits) - 1,\n        }\n    }\n\n    pub fn into_owned(&self) -> Storage<'static> {\n        Storage {\n            registers: self.registers.into_owned(),\n            index_shift: self.index_shift,\n            precision: self.precision,\n            hash_mask: self.hash_mask,\n        }\n    }\n\n    pub fn add_hash(&mut self, hash: u64) {\n        let (idx, count) = self.idx_count_from_hash(hash);\n        self.registers.set_max(idx, count);\n    }\n\n    pub fn add_encoded(&mut self, encoded: crate::sparse::Encoded) {\n        let (idx, count) = self.idx_count_from_encoded(encoded);\n        self.registers.set_max(idx, count);\n    }\n\n    fn idx_count_from_hash(&self, hash: u64) -> (usize, u8) {\n        let idx = hash.extract(63, self.precision);\n        // w in the paper\n        let hash_bits = hash.extract_bits(63 - self.precision, 0);\n        let count = hash_bits.q() - self.precision;\n        (idx as usize, count)\n    }\n\n    fn idx_count_from_encoded(&self, encoded: crate::sparse::Encoded) -> (usize, u8) {\n        let old_idx = encoded.idx();\n        let idx = old_idx >> (25 - self.precision);\n        let count = encoded.count(self.precision);\n        (idx as usize, count)\n    }\n\n    pub fn estimate_count(&self) -> u64 {\n        let num_zeros = self.registers.count_zeroed_registers();\n        let sum: f64 = self\n            .registers\n            .iter()\n            .map(|v| 2.0f64.powi(-(v as i32)))\n            .sum();\n        let m = (1 << self.precision) as f64;\n        let a_m = self.a_m();\n        let e = a_m * m.powi(2) / sum;\n        let e_p = if e <= 5.0 * m {\n            e - self.estimate_bias(e)\n        } else {\n            e\n        };\n\n        let h = if num_zeros != 0 {\n            self.linear_counting(num_zeros as f64)\n        } else {\n            e_p\n        };\n\n        if h <= self.threshold() {\n            h as u64\n        } else {\n            e_p as u64\n        }\n    }\n\n    fn linear_counting(&self, v: f64) -> f64 {\n        let m = (1 << self.precision) as f64;\n        m * (m / v).ln()\n    }\n\n    fn threshold(&self) -> f64 {\n        THRESHOLD_DATA_VEC[self.precision as usize - THRESHOLD_DATA_OFFSET] as f64\n    }\n\n    fn a_m(&self) -> f64 {\n        let size = 1 << self.precision;\n        let m = size as f64;\n        match size {\n            16 => 0.673,\n            32 => 0.697,\n            64 => 0.709,\n            _ => 0.7213 / (1.0 + 1.079 / m),\n        }\n    }\n\n    fn estimate_bias(&self, estimate: f64) -> f64 {\n        use Bounds::*;\n\n        let raw_estimates =\n            RAW_ESTIMATE_DATA_VEC[self.precision as usize - RAW_ESTIMATE_DATA_OFFSET];\n        let bias_data = BIAS_DATA_VEC[self.precision as usize - BIAS_DATA_OFFSET];\n\n        let start = raw_estimates.binary_search_by(|v| v.partial_cmp(&estimate).unwrap());\n        let mut bounds = match start {\n            Ok(i) => return bias_data[i],\n            Err(0) => Right(0),\n            Err(i) if i == raw_estimates.len() => Left(i - 1),\n            Err(i) => Both(i - 1, i),\n        };\n        let mut neighbors = [0; 6];\n        let mut distances = [0.0; 6];\n        for i in 0..6 {\n            let (idx, distance) = bounds.next_closest(estimate, raw_estimates);\n            neighbors[i] = idx;\n            distances[i] = distance;\n        }\n        for distance in &mut distances {\n            *distance = 1.0 / *distance;\n        }\n        let total: f64 = distances.iter().sum();\n        for distance in &mut distances {\n            *distance /= total;\n        }\n\n        let mut value = 0.0;\n        for i in 0..6 {\n            value += distances[i] * bias_data[neighbors[i]];\n        }\n\n        return value;\n\n        enum Bounds {\n            Left(usize),\n            Right(usize),\n            Both(usize, usize),\n        }\n\n        impl Bounds {\n            // find the closet neighbor to `estimate` in `raw_estimates` and update self\n            fn next_closest(&mut self, estimate: f64, raw_estimates: &[f64]) -> (usize, f64) {\n                match self {\n                    Left(i) => {\n                        let idx = *i;\n                        *i -= 1;\n                        (idx, (raw_estimates[*i] - estimate).abs())\n                    }\n                    Right(i) => {\n                        let idx = *i;\n                        *i += 1;\n                        (idx, (raw_estimates[*i] - estimate).abs())\n                    }\n                    Both(l, r) => {\n                        let left_delta = (raw_estimates[*l] - estimate).abs();\n                        let right_delta = (raw_estimates[*r] - estimate).abs();\n                        if right_delta < left_delta {\n                            let idx = *r;\n                            if *r < raw_estimates.len() - 1 {\n                                *r += 1;\n                                return (idx, right_delta);\n                            }\n                            *self = Left(*l);\n                            (idx, right_delta)\n                        } else {\n                            let idx = *l;\n                            if *l > 0 {\n                                *l -= 1;\n                                return (idx, left_delta);\n                            }\n                            *self = Right(*r);\n                            (idx, left_delta)\n                        }\n                    }\n                }\n            }\n        }\n    }\n\n    pub fn merge_in(&mut self, other: &Storage<'_>) {\n        assert!(\n            self.precision == other.precision,\n            \"precision must be equal (left={}, right={})\",\n            self.precision,\n            other.precision\n        );\n\n        assert!(\n            self.registers.bytes().len() == other.registers.bytes().len(),\n            \"registers length must be equal (left={}, right={})\",\n            self.registers.bytes().len(),\n            other.registers.bytes().len(),\n        );\n\n        // TODO this is probably inefficient\n        for (i, r) in other.registers.iter().enumerate() {\n            self.registers.set_max(i, r)\n        }\n    }\n\n    pub fn num_bytes(&self) -> usize {\n        self.registers.byte_len()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use fnv::FnvHasher;\n\n    use crate::sparse::Encoded;\n\n    use super::*;\n\n    use std::{\n        collections::HashSet,\n        hash::{Hash, Hasher},\n    };\n\n    pub fn hash<V: Hash>(val: V) -> u64 {\n        let mut hasher = FnvHasher::default();\n        val.hash(&mut hasher);\n        hasher.finish()\n    }\n\n    #[test]\n    #[should_panic(expected = \"invalid value for precision: 3; must be within [4, 18]\")]\n    fn new_panics_b3() {\n        Storage::new(3);\n    }\n\n    #[test]\n    fn new_works_b4() {\n        Storage::new(4);\n    }\n\n    #[test]\n    fn new_works_b18() {\n        Storage::new(18);\n    }\n\n    #[test]\n    #[should_panic(expected = \"invalid value for precision: 19; must be within [4, 18]\")]\n    fn new_panics_b19() {\n        Storage::new(19);\n    }\n\n    #[test]\n    fn empty() {\n        assert_eq!(Storage::new(8).estimate_count(), 0);\n    }\n\n    #[test]\n    fn add_b4_n1k() {\n        let mut hll = Storage::new(4);\n        for i in 0..1000 {\n            hll.add_hash(hash(i));\n        }\n        // FIXME examine in more detail\n        assert_eq!(hll.estimate_count(), 96);\n    }\n\n    #[test]\n    fn add_b8_n1k() {\n        let mut hll = Storage::new(8);\n        for i in 0..1000 {\n            hll.add_hash(hash(i));\n        }\n        // FIXME examine in more detail\n        assert_eq!(hll.estimate_count(), 430);\n    }\n\n    #[test]\n    fn add_b12_n1k() {\n        let mut hll = Storage::new(12);\n        for i in 0..1000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 1146);\n    }\n\n    #[test]\n    fn add_b16_n1k() {\n        let mut hll = Storage::new(16);\n        for i in 0..1000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 1007);\n    }\n\n    #[test]\n    fn add_b8_n10k() {\n        let mut hll = Storage::new(8);\n        for i in 0..10000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 10536);\n    }\n\n    #[test]\n    fn add_b12_n10k() {\n        let mut hll = Storage::new(12);\n        for i in 0..10000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 11347);\n    }\n\n    #[test]\n    fn add_b16_n10k() {\n        let mut hll = Storage::new(16);\n        for i in 0..10000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 10850);\n    }\n\n    #[test]\n    fn add_b16_n100k() {\n        let mut hll = Storage::new(16);\n        for i in 0..100000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 117304);\n    }\n\n    #[test]\n    fn add_b16_n1m() {\n        let mut hll = Storage::new(16);\n        for i in 0..1000000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 882644);\n    }\n\n    #[test]\n    fn clone() {\n        let mut hll1 = Storage::new(12);\n        for i in 0..500 {\n            hll1.add_hash(hash(i));\n        }\n        let c1a = hll1.estimate_count();\n\n        let hll2 = hll1.clone();\n        assert_eq!(hll2.estimate_count(), c1a);\n\n        for i in 501..1000 {\n            hll1.add_hash(hash(i));\n        }\n        let c1b = hll1.estimate_count();\n        assert_ne!(c1b, c1a);\n        assert_eq!(hll2.estimate_count(), c1a);\n    }\n\n    #[test]\n    fn merge() {\n        let mut hll1 = Storage::new(12);\n        let mut hll2 = Storage::new(12);\n        let mut hll = Storage::new(12);\n        for i in 0..500 {\n            hll.add_hash(hash(i));\n            hll1.add_hash(hash(i));\n        }\n        for i in 501..1000 {\n            hll.add_hash(hash(i));\n            hll2.add_hash(hash(i));\n        }\n        assert_ne!(hll.estimate_count(), hll1.estimate_count());\n        assert_ne!(hll.estimate_count(), hll2.estimate_count());\n\n        hll1.merge_in(&hll2);\n        assert_eq!(hll.estimate_count(), hll1.estimate_count());\n    }\n\n    #[test]\n    #[should_panic(expected = \"precision must be equal (left=5, right=12)\")]\n    fn merge_panics_p() {\n        let mut hll1 = Storage::new(5);\n        let hll2 = Storage::new(12);\n        hll1.merge_in(&hll2);\n    }\n\n    #[test]\n    fn issue_74() {\n        let panic_data = vec![\n            \"ofr-1-1517560282779878449\",\n            \"ofr-1-1517589543534331019\",\n            \"ofr-1-1517590532450550786\",\n            \"ofr-1-1517644560121333465\",\n            \"ofr-1-1517746611185649116\",\n            \"ofr-1-1518051376300950677\",\n            \"ofr-1-1518484387459892414\",\n            \"ofr-1-1518488008830355319\",\n            \"ofr-1-1518488407814571264\",\n            \"ofr-1-1518561818180978525\",\n            \"ofr-1-1518678274740717330\",\n            \"ofr-1-1519461045930165638\",\n            \"ofr-1-1519470647696557288\",\n            \"ofr-1-1519567114956309703\",\n            \"ofr-1-1519653616441755584\",\n            \"ofr-1-1519655049912256356\",\n            \"ofr-1-1520105514088138521\",\n            \"ofr-1-1520294225822221822\",\n            \"ofr-1-1520319017418884884\",\n            \"ofr-1-1520505982893295286\",\n            \"ofr-1-1520553027150677707\",\n            \"ofr-1-1520925550686111649\",\n            \"ofr-1-1520927095122167663\",\n            \"ofr-1-1521290010424640726\",\n            \"ofr-1-1521458659554886917\",\n            \"ofr-1-1521943577454052994\",\n            \"ofr-1-1521971260753839540\",\n            \"ofr-1-1522000670785668758\",\n            \"ofr-1-1522043914876749176\",\n            \"ofr-1-1522206531944580201\",\n            \"ofr-1-1522234960069920034\",\n            \"ofr-1-1522333169901504119\",\n            \"ofr-1-1522363887846294936\",\n            \"ofr-1-1522484446749918495\",\n            \"ofr-1-1522600458059122179\",\n            \"ofr-1-1522687450205783676\",\n            \"ofr-1-1522765602785461678\",\n            \"ofr-1-1522815395559769187\",\n            \"ofr-1-1522839112893465736\",\n            \"ofr-1-1523001178903151627\",\n            \"ofr-1-1523018056414397988\",\n            \"ofr-1-1523096555609261412\",\n            \"ofr-1-1523103371222189143\",\n            \"ofr-1-1523256333918667890\",\n            \"ofr-1-1523270427746895732\",\n            \"ofr-1-1523411745695466681\",\n            \"ofr-1-1523630566301631536\",\n            \"ofr-1-1523839014553388093\",\n            \"ofr-1-1523894230803940925\",\n            \"ofr-1-1523931915564221543\",\n            \"ofr-1-1524104734332815100\",\n            \"ofr-1-1524113364834715372\",\n            \"ofr-1-1524209603273164167\",\n            \"ofr-1-1524276802153219312\",\n            \"ofr-1-1524554894791804305\",\n            \"ofr-1-1524621894100584193\",\n        ];\n\n        let mut hll = Storage::new(4);\n        for entry in &panic_data {\n            hll.add_hash(hash(entry));\n        }\n\n        hll.estimate_count();\n    }\n\n    #[quickcheck]\n    fn quick_16(values: HashSet<u64>) -> quickcheck::TestResult {\n        let mut hll = Storage::new(16);\n        let expected = values.iter().collect::<HashSet<_>>().len() as f64;\n        for value in values {\n            hll.add_hash(value);\n        }\n        let estimated = hll.estimate_count() as f64;\n        let error = 0.01 * expected;\n        // quickcheck instantly finds hash collisions, so we can only check that\n        // we underestimate the cardinality\n        if estimated <= expected + error {\n            return quickcheck::TestResult::passed();\n        }\n        println!(\"got {}, expected {} +- {}\", estimated, expected, error);\n        quickcheck::TestResult::failed()\n    }\n\n    #[cfg(feature = \"flaky_tests\")]\n    #[quickcheck]\n    fn quick_8(values: Vec<u64>) -> quickcheck::TestResult {\n        let mut hll = Storage::new(8);\n        let expected = values.iter().collect::<HashSet<_>>().len() as f64;\n        for value in values {\n            hll.add_hash(value);\n        }\n        let estimated = hll.estimate_count() as f64;\n        let error = 0.10 * expected;\n        // quickcheck instantly finds hash collisions, so we can only check that\n        // we underestimate the cardinality\n        if estimated <= expected + error {\n            return quickcheck::TestResult::passed();\n        }\n        println!(\"got {}, expected {} +- {}\", estimated, expected, error);\n        quickcheck::TestResult::failed()\n    }\n\n    #[quickcheck]\n    fn quick_decode_16(value: u64) -> bool {\n        let hll = Storage::new(8);\n        let from_hash = hll.idx_count_from_hash(value);\n        let from_encoded = hll.idx_count_from_encoded(Encoded::from_hash(value, hll.precision));\n        if from_hash != from_encoded {\n            println!(\n                \"{:#x}, expected {:?}, got {:?}\",\n                value, from_hash, from_encoded\n            );\n            return false;\n        }\n\n        true\n    }\n}\n"
  },
  {
    "path": "crates/hyperloglogplusplus/src/hyperloglog_data.rs",
    "content": "// based on https://github.com/crepererum/pdatastructs.rs/blob/e4f49e6462187700b9a12e8301df9a72a0c6e58c/src/hyperloglog_data.rs\n// and https://goo.gl/iU8Ig\n\n#![allow(clippy::unreadable_literal)]\n\npub(crate) const THRESHOLD_DATA_OFFSET: usize = 4;\npub(crate) const THRESHOLD_DATA_VEC: &[usize] = &[\n    10,     // b = 4\n    20,     // b = 5\n    40,     // b = 6\n    80,     // b = 7\n    220,    // b = 8\n    400,    // b = 9\n    900,    // b = 10\n    1800,   // b = 11\n    3100,   // b = 12\n    6500,   // b = 13\n    11500,  // b = 14\n    20000,  // b = 15\n    50000,  // b = 16\n    120000, // b = 17\n    350000, // b = 18\n];\n\npub(crate) const RAW_ESTIMATE_DATA_OFFSET: usize = 4;\npub(crate) const RAW_ESTIMATE_DATA_VEC: &[&[f64]] = &[\n    // precision 4\n    &[\n        11., 11.717, 12.207, 12.7896, 13.2882, 13.8204, 14.3772, 14.9342, 15.5202, 16.161, 16.7722,\n        17.4636, 18.0396, 18.6766, 19.3566, 20.0454, 20.7936, 21.4856, 22.2666, 22.9946, 23.766,\n        24.4692, 25.3638, 26.0764, 26.7864, 27.7602, 28.4814, 29.433, 30.2926, 31.0664, 31.9996,\n        32.7956, 33.5366, 34.5894, 35.5738, 36.2698, 37.3682, 38.0544, 39.2342, 40.0108, 40.7966,\n        41.9298, 42.8704, 43.6358, 44.5194, 45.773, 46.6772, 47.6174, 48.4888, 49.3304, 50.2506,\n        51.4996, 52.3824, 53.3078, 54.3984, 55.5838, 56.6618, 57.2174, 58.3514, 59.0802, 60.1482,\n        61.0376, 62.3598, 62.8078, 63.9744, 64.914, 65.781, 67.1806, 68.0594, 68.8446, 69.7928,\n        70.8248, 71.8324, 72.8598, 73.6246, 74.7014, 75.393, 76.6708, 77.2394,\n    ],\n    // precision 5\n    &[\n        23., 23.1194, 23.8208, 24.2318, 24.77, 25.2436, 25.7774, 26.2848, 26.8224, 27.3742,\n        27.9336, 28.503, 29.0494, 29.6292, 30.2124, 30.798, 31.367, 31.9728, 32.5944, 33.217,\n        33.8438, 34.3696, 35.0956, 35.7044, 36.324, 37.0668, 37.6698, 38.3644, 39.049, 39.6918,\n        40.4146, 41.082, 41.687, 42.5398, 43.2462, 43.857, 44.6606, 45.4168, 46.1248, 46.9222,\n        47.6804, 48.447, 49.3454, 49.9594, 50.7636, 51.5776, 52.331, 53.19, 53.9676, 54.7564,\n        55.5314, 56.4442, 57.3708, 57.9774, 58.9624, 59.8796, 60.755, 61.472, 62.2076, 63.1024,\n        63.8908, 64.7338, 65.7728, 66.629, 67.413, 68.3266, 69.1524, 70.2642, 71.1806, 72.0566,\n        72.9192, 73.7598, 74.3516, 75.5802, 76.4386, 77.4916, 78.1524, 79.1892, 79.8414, 80.8798,\n        81.8376, 82.4698, 83.7656, 84.331, 85.5914, 86.6012, 87.7016, 88.5582, 89.3394, 90.3544,\n        91.4912, 92.308, 93.3552, 93.9746, 95.2052, 95.727, 97.1322, 98.3944, 98.7588, 100.242,\n        101.1914, 102.2538, 102.8776, 103.6292, 105.1932, 105.9152, 107.0868, 107.6728, 108.7144,\n        110.3114, 110.8716, 111.245, 112.7908, 113.7064, 114.636, 115.7464, 116.1788, 117.7464,\n        118.4896, 119.6166, 120.5082, 121.7798, 122.9028, 123.4426, 124.8854, 125.705, 126.4652,\n        128.3464, 128.3462, 130.0398, 131.0342, 131.0042, 132.4766, 133.511, 134.7252, 135.425,\n        136.5172, 138.0572, 138.6694, 139.3712, 140.8598, 141.4594, 142.554, 143.4006, 144.7374,\n        146.1634, 146.8994, 147.605, 147.9304, 149.1636, 150.2468, 151.5876, 152.2096, 153.7032,\n        154.7146, 155.807, 156.9228, 157.0372, 158.5852,\n    ],\n    // precision 6\n    &[\n        46., 46.1902, 47.271, 47.8358, 48.8142, 49.2854, 50.317, 51.354, 51.8924, 52.9436, 53.4596,\n        54.5262, 55.6248, 56.1574, 57.2822, 57.837, 58.9636, 60.074, 60.7042, 61.7976, 62.4772,\n        63.6564, 64.7942, 65.5004, 66.686, 67.291, 68.5672, 69.8556, 70.4982, 71.8204, 72.4252,\n        73.7744, 75.0786, 75.8344, 77.0294, 77.8098, 79.0794, 80.5732, 81.1878, 82.5648, 83.2902,\n        84.6784, 85.3352, 86.8946, 88.3712, 89.0852, 90.499, 91.2686, 92.6844, 94.2234, 94.9732,\n        96.3356, 97.2286, 98.7262, 100.3284, 101.1048, 102.5962, 103.3562, 105.1272, 106.4184,\n        107.4974, 109.0822, 109.856, 111.48, 113.2834, 114.0208, 115.637, 116.5174, 118.0576,\n        119.7476, 120.427, 122.1326, 123.2372, 125.2788, 126.6776, 127.7926, 129.1952, 129.9564,\n        131.6454, 133.87, 134.5428, 136.2, 137.0294, 138.6278, 139.6782, 141.792, 143.3516,\n        144.2832, 146.0394, 147.0748, 148.4912, 150.849, 151.696, 153.5404, 154.073, 156.3714,\n        157.7216, 158.7328, 160.4208, 161.4184, 163.9424, 165.2772, 166.411, 168.1308, 168.769,\n        170.9258, 172.6828, 173.7502, 175.706, 176.3886, 179.0186, 180.4518, 181.927, 183.4172,\n        184.4114, 186.033, 188.5124, 189.5564, 191.6008, 192.4172, 193.8044, 194.997, 197.4548,\n        198.8948, 200.2346, 202.3086, 203.1548, 204.8842, 206.6508, 206.6772, 209.7254, 210.4752,\n        212.7228, 214.6614, 215.1676, 217.793, 218.0006, 219.9052, 221.66, 223.5588, 225.1636,\n        225.6882, 227.7126, 229.4502, 231.1978, 232.9756, 233.1654, 236.727, 238.1974, 237.7474,\n        241.1346, 242.3048, 244.1948, 245.3134, 246.879, 249.1204, 249.853, 252.6792, 253.857,\n        254.4486, 257.2362, 257.9534, 260.0286, 260.5632, 262.663, 264.723, 265.7566, 267.2566,\n        267.1624, 270.62, 272.8216, 273.2166, 275.2056, 276.2202, 278.3726, 280.3344, 281.9284,\n        283.9728, 284.1924, 286.4872, 287.587, 289.807, 291.1206, 292.769, 294.8708, 296.665,\n        297.1182, 299.4012, 300.6352, 302.1354, 304.1756, 306.1606, 307.3462, 308.5214, 309.4134,\n        310.8352, 313.9684, 315.837, 316.7796, 318.9858,\n    ],\n    // precision 7\n    &[\n        92., 93.4934, 94.9758, 96.4574, 97.9718, 99.4954, 101.5302, 103.0756, 104.6374, 106.1782,\n        107.7888, 109.9522, 111.592, 113.2532, 114.9086, 116.5938, 118.9474, 120.6796, 122.4394,\n        124.2176, 125.9768, 128.4214, 130.2528, 132.0102, 133.8658, 135.7278, 138.3044, 140.1316,\n        142.093, 144.0032, 145.9092, 148.6306, 150.5294, 152.5756, 154.6508, 156.662, 159.552,\n        161.3724, 163.617, 165.5754, 167.7872, 169.8444, 172.7988, 174.8606, 177.2118, 179.3566,\n        181.4476, 184.5882, 186.6816, 189.0824, 191.0258, 193.6048, 196.4436, 198.7274, 200.957,\n        203.147, 205.4364, 208.7592, 211.3386, 213.781, 215.8028, 218.656, 221.6544, 223.996,\n        226.4718, 229.1544, 231.6098, 234.5956, 237.0616, 239.5758, 242.4878, 244.5244, 248.2146,\n        250.724, 252.8722, 255.5198, 258.0414, 261.941, 264.9048, 266.87, 269.4304, 272.028,\n        274.4708, 278.37, 281.0624, 283.4668, 286.5532, 289.4352, 293.2564, 295.2744, 298.2118,\n        300.7472, 304.1456, 307.2928, 309.7504, 312.5528, 315.979, 318.2102, 322.1834, 324.3494,\n        327.325, 330.6614, 332.903, 337.2544, 339.9042, 343.215, 345.2864, 348.0814, 352.6764,\n        355.301, 357.139, 360.658, 363.1732, 366.5902, 369.9538, 373.0828, 375.922, 378.9902,\n        382.7328, 386.4538, 388.1136, 391.2234, 394.0878, 396.708, 401.1556, 404.1852, 406.6372,\n        409.6822, 412.7796, 416.6078, 418.4916, 422.131, 424.5376, 428.1988, 432.211, 434.4502,\n        438.5282, 440.912, 444.0448, 447.7432, 450.8524, 453.7988, 456.7858, 458.8868, 463.9886,\n        466.5064, 468.9124, 472.6616, 475.4682, 478.582, 481.304, 485.2738, 488.6894, 490.329,\n        496.106, 497.6908, 501.1374, 504.5322, 506.8848, 510.3324, 513.4512, 516.179, 520.4412,\n        522.6066, 526.167, 528.7794, 533.379, 536.067, 538.46, 542.9116, 545.692, 547.9546,\n        552.493, 555.2722, 557.335, 562.449, 564.2014, 569.0738, 571.0974, 574.8564, 578.2996,\n        581.409, 583.9704, 585.8098, 589.6528, 594.5998, 595.958, 600.068, 603.3278, 608.2016,\n        609.9632, 612.864, 615.43, 620.7794, 621.272, 625.8644, 629.206, 633.219, 634.5154,\n        638.6102,\n    ],\n    // precision 8\n    &[\n        184.2152, 187.2454, 190.2096, 193.6652, 196.6312, 199.6822, 203.249, 206.3296, 210.0038,\n        213.2074, 216.4612, 220.27, 223.5178, 227.4412, 230.8032, 234.1634, 238.1688, 241.6074,\n        245.6946, 249.2664, 252.8228, 257.0432, 260.6824, 264.9464, 268.6268, 272.2626, 276.8376,\n        280.4034, 284.8956, 288.8522, 292.7638, 297.3552, 301.3556, 305.7526, 309.9292, 313.8954,\n        318.8198, 322.7668, 327.298, 331.6688, 335.9466, 340.9746, 345.1672, 349.3474, 354.3028,\n        358.8912, 364.114, 368.4646, 372.9744, 378.4092, 382.6022, 387.843, 392.5684, 397.1652,\n        402.5426, 407.4152, 412.5388, 417.3592, 422.1366, 427.486, 432.3918, 437.5076, 442.509,\n        447.3834, 453.3498, 458.0668, 463.7346, 469.1228, 473.4528, 479.7, 484.644, 491.0518,\n        495.5774, 500.9068, 506.432, 512.1666, 517.434, 522.6644, 527.4894, 533.6312, 538.3804,\n        544.292, 550.5496, 556.0234, 562.8206, 566.6146, 572.4188, 579.117, 583.6762, 590.6576,\n        595.7864, 601.509, 607.5334, 612.9204, 619.772, 624.2924, 630.8654, 636.1836, 642.745,\n        649.1316, 655.0386, 660.0136, 666.6342, 671.6196, 678.1866, 684.4282, 689.3324, 695.4794,\n        702.5038, 708.129, 713.528, 720.3204, 726.463, 732.7928, 739.123, 744.7418, 751.2192,\n        756.5102, 762.6066, 769.0184, 775.2224, 781.4014, 787.7618, 794.1436, 798.6506, 805.6378,\n        811.766, 819.7514, 824.5776, 828.7322, 837.8048, 843.6302, 849.9336, 854.4798, 861.3388,\n        867.9894, 873.8196, 880.3136, 886.2308, 892.4588, 899.0816, 905.4076, 912.0064, 917.3878,\n        923.619, 929.998, 937.3482, 943.9506, 947.991, 955.1144, 962.203, 968.8222, 975.7324,\n        981.7826, 988.7666, 994.2648, 1000.3128, 1007.4082, 1013.7536, 1020.3376, 1026.7156,\n        1031.7478, 1037.4292, 1045.393, 1051.2278, 1058.3434, 1062.8726, 1071.884, 1076.806,\n        1082.9176, 1089.1678, 1095.5032, 1102.525, 1107.2264, 1115.315, 1120.93, 1127.252,\n        1134.1496, 1139.0408, 1147.5448, 1153.3296, 1158.1974, 1166.5262, 1174.3328, 1175.657,\n        1184.4222, 1190.9172, 1197.1292, 1204.4606, 1210.4578, 1218.8728, 1225.3336, 1226.6592,\n        1236.5768, 1241.363, 1249.4074, 1254.6566, 1260.8014, 1266.5454, 1274.5192,\n    ],\n    // precision 9\n    &[\n        369., 374.8294, 381.2452, 387.6698, 394.1464, 400.2024, 406.8782, 413.6598, 420.462,\n        427.2826, 433.7102, 440.7416, 447.9366, 455.1046, 462.285, 469.0668, 476.306, 483.8448,\n        491.301, 498.9886, 506.2422, 513.8138, 521.7074, 529.7428, 537.8402, 545.1664, 553.3534,\n        561.594, 569.6886, 577.7876, 585.65, 594.228, 602.8036, 611.1666, 620.0818, 628.0824,\n        637.2574, 646.302, 655.1644, 664.0056, 672.3802, 681.7192, 690.5234, 700.2084, 708.831,\n        718.485, 728.1112, 737.4764, 746.76, 756.3368, 766.5538, 775.5058, 785.2646, 795.5902,\n        804.3818, 814.8998, 824.9532, 835.2062, 845.2798, 854.4728, 864.9582, 875.3292, 886.171,\n        896.781, 906.5716, 916.7048, 927.5322, 937.875, 949.3972, 958.3464, 969.7274, 980.2834,\n        992.1444, 1003.4264, 1013.0166, 1024.018, 1035.0438, 1046.34, 1057.6856, 1068.9836,\n        1079.0312, 1091.677, 1102.3188, 1113.4846, 1124.4424, 1135.739, 1147.1488, 1158.9202,\n        1169.406, 1181.5342, 1193.2834, 1203.8954, 1216.3286, 1226.2146, 1239.6684, 1251.9946,\n        1262.123, 1275.4338, 1285.7378, 1296.076, 1308.9692, 1320.4964, 1333.0998, 1343.9864,\n        1357.7754, 1368.3208, 1380.4838, 1392.7388, 1406.0758, 1416.9098, 1428.9728, 1440.9228,\n        1453.9292, 1462.617, 1476.05, 1490.2996, 1500.6128, 1513.7392, 1524.5174, 1536.6322,\n        1548.2584, 1562.3766, 1572.423, 1587.1232, 1596.5164, 1610.5938, 1622.5972, 1633.1222,\n        1647.7674, 1658.5044, 1671.57, 1683.7044, 1695.4142, 1708.7102, 1720.6094, 1732.6522,\n        1747.841, 1756.4072, 1769.9786, 1782.3276, 1797.5216, 1808.3186, 1819.0694, 1834.354,\n        1844.575, 1856.2808, 1871.1288, 1880.7852, 1893.9622, 1906.3418, 1920.6548, 1932.9302,\n        1945.8584, 1955.473, 1968.8248, 1980.6446, 1995.9598, 2008.349, 2019.8556, 2033.0334,\n        2044.0206, 2059.3956, 2069.9174, 2082.6084, 2093.7036, 2106.6108, 2118.9124, 2132.301,\n        2144.7628, 2159.8422, 2171.0212, 2183.101, 2193.5112, 2208.052, 2221.3194, 2233.3282,\n        2247.295, 2257.7222, 2273.342, 2286.5638, 2299.6786, 2310.8114, 2322.3312, 2335.516,\n        2349.874, 2363.5968, 2373.865, 2387.1918, 2401.8328, 2414.8496, 2424.544, 2436.7592,\n        2447.1682, 2464.1958, 2474.3438, 2489.0006, 2497.4526, 2513.6586, 2527.19, 2540.7028,\n        2553.768,\n    ],\n    // precision 10\n    &[\n        738.1256, 750.4234, 763.1064, 775.4732, 788.4636, 801.0644, 814.488, 827.9654, 841.0832,\n        854.7864, 868.1992, 882.2176, 896.5228, 910.1716, 924.7752, 938.899, 953.6126, 968.6492,\n        982.9474, 998.5214, 1013.1064, 1028.6364, 1044.2468, 1059.4588, 1075.3832, 1091.0584,\n        1106.8606, 1123.3868, 1139.5062, 1156.1862, 1172.463, 1189.339, 1206.1936, 1223.1292,\n        1240.1854, 1257.2908, 1275.3324, 1292.8518, 1310.5204, 1328.4854, 1345.9318, 1364.552,\n        1381.4658, 1400.4256, 1419.849, 1438.152, 1456.8956, 1474.8792, 1494.118, 1513.62,\n        1532.5132, 1551.9322, 1570.7726, 1590.6086, 1610.5332, 1630.5918, 1650.4294, 1669.7662,\n        1690.4106, 1710.7338, 1730.9012, 1750.4486, 1770.1556, 1791.6338, 1812.7312, 1833.6264,\n        1853.9526, 1874.8742, 1896.8326, 1918.1966, 1939.5594, 1961.07, 1983.037, 2003.1804,\n        2026.071, 2047.4884, 2070.0848, 2091.2944, 2114.333, 2135.9626, 2158.2902, 2181.0814,\n        2202.0334, 2224.4832, 2246.39, 2269.7202, 2292.1714, 2314.2358, 2338.9346, 2360.891,\n        2384.0264, 2408.3834, 2430.1544, 2454.8684, 2476.9896, 2501.4368, 2522.8702, 2548.0408,\n        2570.6738, 2593.5208, 2617.0158, 2640.2302, 2664.0962, 2687.4986, 2714.2588, 2735.3914,\n        2759.6244, 2781.8378, 2808.0072, 2830.6516, 2856.2454, 2877.2136, 2903.4546, 2926.785,\n        2951.2294, 2976.468, 3000.867, 3023.6508, 3049.91, 3073.5984, 3098.162, 3121.5564,\n        3146.2328, 3170.9484, 3195.5902, 3221.3346, 3242.7032, 3271.6112, 3296.5546, 3317.7376,\n        3345.072, 3369.9518, 3394.326, 3418.1818, 3444.6926, 3469.086, 3494.2754, 3517.8698,\n        3544.248, 3565.3768, 3588.7234, 3616.979, 3643.7504, 3668.6812, 3695.72, 3719.7392,\n        3742.6224, 3770.4456, 3795.6602, 3819.9058, 3844.002, 3869.517, 3895.6824, 3920.8622,\n        3947.1364, 3973.985, 3995.4772, 4021.62, 4046.628, 4074.65, 4096.2256, 4121.831, 4146.6406,\n        4173.276, 4195.0744, 4223.9696, 4251.3708, 4272.9966, 4300.8046, 4326.302, 4353.1248,\n        4374.312, 4403.0322, 4426.819, 4450.0598, 4478.5206, 4504.8116, 4528.8928, 4553.9584,\n        4578.8712, 4603.8384, 4632.3872, 4655.5128, 4675.821, 4704.6222, 4731.9862, 4755.4174,\n        4781.2628, 4804.332, 4832.3048, 4862.8752, 4883.4148, 4906.9544, 4935.3516, 4954.3532,\n        4984.0248, 5011.217, 5035.3258, 5057.3672, 5084.1828,\n    ],\n    // precision 11\n    &[\n        1477., 1501.6014, 1526.5802, 1551.7942, 1577.3042, 1603.2062, 1629.8402, 1656.2292,\n        1682.9462, 1709.9926, 1737.3026, 1765.4252, 1793.0578, 1821.6092, 1849.626, 1878.5568,\n        1908.527, 1937.5154, 1967.1874, 1997.3878, 2027.37, 2058.1972, 2089.5728, 2120.1012,\n        2151.9668, 2183.292, 2216.0772, 2247.8578, 2280.6562, 2313.041, 2345.714, 2380.3112,\n        2414.1806, 2447.9854, 2481.656, 2516.346, 2551.5154, 2586.8378, 2621.7448, 2656.6722,\n        2693.5722, 2729.1462, 2765.4124, 2802.8728, 2838.898, 2876.408, 2913.4926, 2951.4938,\n        2989.6776, 3026.282, 3065.7704, 3104.1012, 3143.7388, 3181.6876, 3221.1872, 3261.5048,\n        3300.0214, 3339.806, 3381.409, 3421.4144, 3461.4294, 3502.2286, 3544.651, 3586.6156,\n        3627.337, 3670.083, 3711.1538, 3753.5094, 3797.01, 3838.6686, 3882.1678, 3922.8116,\n        3967.9978, 4009.9204, 4054.3286, 4097.5706, 4140.6014, 4185.544, 4229.5976, 4274.583,\n        4316.9438, 4361.672, 4406.2786, 4451.8628, 4496.1834, 4543.505, 4589.1816, 4632.5188,\n        4678.2294, 4724.8908, 4769.0194, 4817.052, 4861.4588, 4910.1596, 4956.4344, 5002.5238,\n        5048.13, 5093.6374, 5142.8162, 5187.7894, 5237.3984, 5285.6078, 5331.0858, 5379.1036,\n        5428.6258, 5474.6018, 5522.7618, 5571.5822, 5618.59, 5667.9992, 5714.88, 5763.454,\n        5808.6982, 5860.3644, 5910.2914, 5953.571, 6005.9232, 6055.1914, 6104.5882, 6154.5702,\n        6199.7036, 6251.1764, 6298.7596, 6350.0302, 6398.061, 6448.4694, 6495.933, 6548.0474,\n        6597.7166, 6646.9416, 6695.9208, 6742.6328, 6793.5276, 6842.1934, 6894.2372, 6945.3864,\n        6996.9228, 7044.2372, 7094.1374, 7142.2272, 7192.2942, 7238.8338, 7288.9006, 7344.0908,\n        7394.8544, 7443.5176, 7490.4148, 7542.9314, 7595.6738, 7641.9878, 7694.3688, 7743.0448,\n        7797.522, 7845.53, 7899.594, 7950.3132, 7996.455, 8050.9442, 8092.9114, 8153.1374,\n        8197.4472, 8252.8278, 8301.8728, 8348.6776, 8401.4698, 8453.551, 8504.6598, 8553.8944,\n        8604.1276, 8657.6514, 8710.3062, 8758.908, 8807.8706, 8862.1702, 8910.4668, 8960.77,\n        9007.2766, 9063.164, 9121.0534, 9164.1354, 9218.1594, 9267.767, 9319.0594, 9372.155,\n        9419.7126, 9474.3722, 9520.1338, 9572.368, 9622.7702, 9675.8448, 9726.5396, 9778.7378,\n        9827.6554, 9878.1922, 9928.7782, 9978.3984, 10026.578, 10076.5626, 10137.1618, 10177.5244,\n        10229.9176,\n    ],\n    // precision 12\n    &[\n        2954., 3003.4782, 3053.3568, 3104.3666, 3155.324, 3206.9598, 3259.648, 3312.539, 3366.1474,\n        3420.2576, 3474.8376, 3530.6076, 3586.451, 3643.38, 3700.4104, 3757.5638, 3815.9676,\n        3875.193, 3934.838, 3994.8548, 4055.018, 4117.1742, 4178.4482, 4241.1294, 4304.4776,\n        4367.4044, 4431.8724, 4496.3732, 4561.4304, 4627.5326, 4693.949, 4761.5532, 4828.7256,\n        4897.6182, 4965.5186, 5034.4528, 5104.865, 5174.7164, 5244.6828, 5316.6708, 5387.8312,\n        5459.9036, 5532.476, 5604.8652, 5679.6718, 5753.757, 5830.2072, 5905.2828, 5980.0434,\n        6056.6264, 6134.3192, 6211.5746, 6290.0816, 6367.1176, 6447.9796, 6526.5576, 6606.1858,\n        6686.9144, 6766.1142, 6847.0818, 6927.9664, 7010.9096, 7091.0816, 7175.3962, 7260.3454,\n        7344.018, 7426.4214, 7511.3106, 7596.0686, 7679.8094, 7765.818, 7852.4248, 7936.834,\n        8022.363, 8109.5066, 8200.4554, 8288.5832, 8373.366, 8463.4808, 8549.7682, 8642.0522,\n        8728.3288, 8820.9528, 8907.727, 9001.0794, 9091.2522, 9179.988, 9269.852, 9362.6394,\n        9453.642, 9546.9024, 9640.6616, 9732.6622, 9824.3254, 9917.7484, 10007.9392, 10106.7508,\n        10196.2152, 10289.8114, 10383.5494, 10482.3064, 10576.8734, 10668.7872, 10764.7156,\n        10862.0196, 10952.793, 11049.9748, 11146.0702, 11241.4492, 11339.2772, 11434.2336,\n        11530.741, 11627.6136, 11726.311, 11821.5964, 11918.837, 12015.3724, 12113.0162,\n        12213.0424, 12306.9804, 12408.4518, 12504.8968, 12604.586, 12700.9332, 12798.705,\n        12898.5142, 12997.0488, 13094.788, 13198.475, 13292.7764, 13392.9698, 13486.8574,\n        13590.1616, 13686.5838, 13783.6264, 13887.2638, 13992.0978, 14081.0844, 14189.9956,\n        14280.0912, 14382.4956, 14486.4384, 14588.1082, 14686.2392, 14782.276, 14888.0284,\n        14985.1864, 15088.8596, 15187.0998, 15285.027, 15383.6694, 15495.8266, 15591.3736,\n        15694.2008, 15790.3246, 15898.4116, 15997.4522, 16095.5014, 16198.8514, 16291.7492,\n        16402.6424, 16499.1266, 16606.2436, 16697.7186, 16796.3946, 16902.3376, 17005.7672,\n        17100.814, 17206.8282, 17305.8262, 17416.0744, 17508.4092, 17617.0178, 17715.4554,\n        17816.758, 17920.1748, 18012.9236, 18119.7984, 18223.2248, 18324.2482, 18426.6276,\n        18525.0932, 18629.8976, 18733.2588, 18831.0466, 18940.1366, 19032.2696, 19131.729,\n        19243.4864, 19349.6932, 19442.866, 19547.9448, 19653.2798, 19754.4034, 19854.0692,\n        19965.1224, 20065.1774, 20158.2212, 20253.353, 20366.3264, 20463.22,\n    ],\n    // precision 13\n    &[\n        5908.5052, 6007.2672, 6107.347, 6208.5794, 6311.2622, 6414.5514, 6519.3376, 6625.6952,\n        6732.5988, 6841.3552, 6950.5972, 7061.3082, 7173.5646, 7287.109, 7401.8216, 7516.4344,\n        7633.3802, 7751.2962, 7870.3784, 7990.292, 8110.79, 8233.4574, 8356.6036, 8482.2712,\n        8607.7708, 8735.099, 8863.1858, 8993.4746, 9123.8496, 9255.6794, 9388.5448, 9522.7516,\n        9657.3106, 9792.6094, 9930.5642, 10068.794, 10206.7256, 10347.81, 10490.3196, 10632.0778,\n        10775.9916, 10920.4662, 11066.124, 11213.073, 11358.0362, 11508.1006, 11659.1716,\n        11808.7514, 11959.4884, 12112.1314, 12265.037, 12420.3756, 12578.933, 12734.311,\n        12890.0006, 13047.2144, 13207.3096, 13368.5144, 13528.024, 13689.847, 13852.7528,\n        14018.3168, 14180.5372, 14346.9668, 14513.5074, 14677.867, 14846.2186, 15017.4186,\n        15184.9716, 15356.339, 15529.2972, 15697.3578, 15871.8686, 16042.187, 16216.4094,\n        16389.4188, 16565.9126, 16742.3272, 16919.0042, 17094.7592, 17273.965, 17451.8342,\n        17634.4254, 17810.5984, 17988.9242, 18171.051, 18354.7938, 18539.466, 18721.0408,\n        18904.9972, 19081.867, 19271.9118, 19451.8694, 19637.9816, 19821.2922, 20013.1292,\n        20199.3858, 20387.8726, 20572.9514, 20770.7764, 20955.1714, 21144.751, 21329.9952,\n        21520.709, 21712.7016, 21906.3868, 22096.2626, 22286.0524, 22475.051, 22665.5098,\n        22862.8492, 23055.5294, 23249.6138, 23437.848, 23636.273, 23826.093, 24020.3296,\n        24213.3896, 24411.7392, 24602.9614, 24805.7952, 24998.1552, 25193.9588, 25389.0166,\n        25585.8392, 25780.6976, 25981.2728, 26175.977, 26376.5252, 26570.1964, 26773.387,\n        26962.9812, 27163.0586, 27368.164, 27565.0534, 27758.7428, 27961.1276, 28163.2324,\n        28362.3816, 28565.7668, 28758.644, 28956.9768, 29163.4722, 29354.7026, 29561.1186,\n        29767.9948, 29959.9986, 30164.0492, 30366.9818, 30562.5338, 30762.9928, 30976.1592,\n        31166.274, 31376.722, 31570.3734, 31770.809, 31974.8934, 32179.5286, 32387.5442,\n        32582.3504, 32794.076, 32989.9528, 33191.842, 33392.4684, 33595.659, 33801.8672,\n        34000.3414, 34200.0922, 34402.6792, 34610.0638, 34804.0084, 35011.13, 35218.669,\n        35418.6634, 35619.0792, 35830.6534, 36028.4966, 36229.7902, 36438.6422, 36630.7764,\n        36833.3102, 37048.6728, 37247.3916, 37453.5904, 37669.3614, 37854.5526, 38059.305,\n        38268.0936, 38470.2516, 38674.7064, 38876.167, 39068.3794, 39281.9144, 39492.8566,\n        39684.8628, 39898.4108, 40093.1836, 40297.6858, 40489.7086, 40717.2424,\n    ],\n    // precision 14\n    &[\n        11817.475, 12015.0046, 12215.3792, 12417.7504, 12623.1814, 12830.0086, 13040.0072,\n        13252.503, 13466.178, 13683.2738, 13902.0344, 14123.9798, 14347.394, 14573.7784,\n        14802.6894, 15033.6824, 15266.9134, 15502.8624, 15741.4944, 15980.7956, 16223.8916,\n        16468.6316, 16715.733, 16965.5726, 17217.204, 17470.666, 17727.8516, 17986.7886,\n        18247.6902, 18510.9632, 18775.304, 19044.7486, 19314.4408, 19587.202, 19862.2576,\n        20135.924, 20417.0324, 20697.9788, 20979.6112, 21265.0274, 21550.723, 21841.6906,\n        22132.162, 22428.1406, 22722.127, 23020.5606, 23319.7394, 23620.4014, 23925.2728,\n        24226.9224, 24535.581, 24845.505, 25155.9618, 25470.3828, 25785.9702, 26103.7764,\n        26420.4132, 26742.0186, 27062.8852, 27388.415, 27714.6024, 28042.296, 28365.4494,\n        28701.1526, 29031.8008, 29364.2156, 29704.497, 30037.1458, 30380.111, 30723.8168,\n        31059.5114, 31404.9498, 31751.6752, 32095.2686, 32444.7792, 32794.767, 33145.204,\n        33498.4226, 33847.6502, 34209.006, 34560.849, 34919.4838, 35274.9778, 35635.1322,\n        35996.3266, 36359.1394, 36722.8266, 37082.8516, 37447.7354, 37815.9606, 38191.0692,\n        38559.4106, 38924.8112, 39294.6726, 39663.973, 40042.261, 40416.2036, 40779.2036,\n        41161.6436, 41540.9014, 41921.1998, 42294.7698, 42678.5264, 43061.3464, 43432.375,\n        43818.432, 44198.6598, 44583.0138, 44970.4794, 45353.924, 45729.858, 46118.2224,\n        46511.5724, 46900.7386, 47280.6964, 47668.1472, 48055.6796, 48446.9436, 48838.7146,\n        49217.7296, 49613.7796, 50010.7508, 50410.0208, 50793.7886, 51190.2456, 51583.1882,\n        51971.0796, 52376.5338, 52763.319, 53165.5534, 53556.5594, 53948.2702, 54346.352,\n        54748.7914, 55138.577, 55543.4824, 55941.1748, 56333.7746, 56745.1552, 57142.7944,\n        57545.2236, 57935.9956, 58348.5268, 58737.5474, 59158.5962, 59542.6896, 59958.8004,\n        60349.3788, 60755.0212, 61147.6144, 61548.194, 61946.0696, 62348.6042, 62763.603,\n        63162.781, 63560.635, 63974.3482, 64366.4908, 64771.5876, 65176.7346, 65597.3916,\n        65995.915, 66394.0384, 66822.9396, 67203.6336, 67612.2032, 68019.0078, 68420.0388,\n        68821.22, 69235.8388, 69640.0724, 70055.155, 70466.357, 70863.4266, 71276.2482, 71677.0306,\n        72080.2006, 72493.0214, 72893.5952, 73314.5856, 73714.9852, 74125.3022, 74521.2122,\n        74933.6814, 75341.5904, 75743.0244, 76166.0278, 76572.1322, 76973.1028, 77381.6284,\n        77800.6092, 78189.328, 78607.0962, 79012.2508, 79407.8358, 79825.725, 80238.701, 80646.891,\n        81035.6436, 81460.0448, 81876.3884,\n    ],\n    // precision 15\n    &[\n        23635.0036,\n        24030.8034,\n        24431.4744,\n        24837.1524,\n        25246.7928,\n        25661.326,\n        26081.3532,\n        26505.2806,\n        26933.9892,\n        27367.7098,\n        27805.318,\n        28248.799,\n        28696.4382,\n        29148.8244,\n        29605.5138,\n        30066.8668,\n        30534.2344,\n        31006.32,\n        31480.778,\n        31962.2418,\n        32447.3324,\n        32938.0232,\n        33432.731,\n        33930.728,\n        34433.9896,\n        34944.1402,\n        35457.5588,\n        35974.5958,\n        36497.3296,\n        37021.9096,\n        37554.326,\n        38088.0826,\n        38628.8816,\n        39171.3192,\n        39723.2326,\n        40274.5554,\n        40832.3142,\n        41390.613,\n        41959.5908,\n        42532.5466,\n        43102.0344,\n        43683.5072,\n        44266.694,\n        44851.2822,\n        45440.7862,\n        46038.0586,\n        46640.3164,\n        47241.064,\n        47846.155,\n        48454.7396,\n        49076.9168,\n        49692.542,\n        50317.4778,\n        50939.65,\n        51572.5596,\n        52210.2906,\n        52843.7396,\n        53481.3996,\n        54127.236,\n        54770.406,\n        55422.6598,\n        56078.7958,\n        56736.7174,\n        57397.6784,\n        58064.5784,\n        58730.308,\n        59404.9784,\n        60077.0864,\n        60751.9158,\n        61444.1386,\n        62115.817,\n        62808.7742,\n        63501.4774,\n        64187.5454,\n        64883.6622,\n        65582.7468,\n        66274.5318,\n        66976.9276,\n        67688.7764,\n        68402.138,\n        69109.6274,\n        69822.9706,\n        70543.6108,\n        71265.5202,\n        71983.3848,\n        72708.4656,\n        73433.384,\n        74158.4664,\n        74896.4868,\n        75620.9564,\n        76362.1434,\n        77098.3204,\n        77835.7662,\n        78582.6114,\n        79323.9902,\n        80067.8658,\n        80814.9246,\n        81567.0136,\n        82310.8536,\n        83061.9952,\n        83821.4096,\n        84580.8608,\n        85335.547,\n        86092.5802,\n        86851.6506,\n        87612.311,\n        88381.2016,\n        89146.3296,\n        89907.8974,\n        90676.846,\n        91451.4152,\n        92224.5518,\n        92995.8686,\n        93763.5066,\n        94551.2796,\n        95315.1944,\n        96096.1806,\n        96881.0918,\n        97665.679,\n        98442.68,\n        99229.3002,\n        100011.0994,\n        100790.6386,\n        101580.1564,\n        102377.7484,\n        103152.1392,\n        103944.2712,\n        104730.216,\n        105528.6336,\n        106324.9398,\n        107117.6706,\n        107890.3988,\n        108695.2266,\n        109485.238,\n        110294.7876,\n        111075.0958,\n        111878.0496,\n        112695.2864,\n        113464.5486,\n        114270.0474,\n        115068.608,\n        115884.3626,\n        116673.2588,\n        117483.3716,\n        118275.097,\n        119085.4092,\n        119879.2808,\n        120687.5868,\n        121499.9944,\n        122284.916,\n        123095.9254,\n        123912.5038,\n        124709.0454,\n        125503.7182,\n        126323.259,\n        127138.9412,\n        127943.8294,\n        128755.646,\n        129556.5354,\n        130375.3298,\n        131161.4734,\n        131971.1962,\n        132787.5458,\n        133588.1056,\n        134431.351,\n        135220.2906,\n        136023.398,\n        136846.6558,\n        137667.0004,\n        138463.663,\n        139283.7154,\n        140074.6146,\n        140901.3072,\n        141721.8548,\n        142543.2322,\n        143356.1096,\n        144173.7412,\n        144973.0948,\n        145794.3162,\n        146609.5714,\n        147420.003,\n        148237.9784,\n        149050.5696,\n        149854.761,\n        150663.1966,\n        151494.0754,\n        152313.1416,\n        153112.6902,\n        153935.7206,\n        154746.9262,\n        155559.547,\n        156401.9746,\n        157228.7036,\n        158008.7254,\n        158820.75,\n        159646.9184,\n        160470.4458,\n        161279.5348,\n        162093.3114,\n        162918.542,\n        163729.2842,\n    ],\n    // precision 16\n    &[\n        47271.,\n        48062.3584,\n        48862.7074,\n        49673.152,\n        50492.8416,\n        51322.9514,\n        52161.03,\n        53009.407,\n        53867.6348,\n        54734.206,\n        55610.5144,\n        56496.2096,\n        57390.795,\n        58297.268,\n        59210.6448,\n        60134.665,\n        61068.0248,\n        62010.4472,\n        62962.5204,\n        63923.5742,\n        64895.0194,\n        65876.4182,\n        66862.6136,\n        67862.6968,\n        68868.8908,\n        69882.8544,\n        70911.271,\n        71944.0924,\n        72990.0326,\n        74040.692,\n        75100.6336,\n        76174.7826,\n        77252.5998,\n        78340.2974,\n        79438.2572,\n        80545.4976,\n        81657.2796,\n        82784.6336,\n        83915.515,\n        85059.7362,\n        86205.9368,\n        87364.4424,\n        88530.3358,\n        89707.3744,\n        90885.9638,\n        92080.197,\n        93275.5738,\n        94479.391,\n        95695.918,\n        96919.2236,\n        98148.4602,\n        99382.3474,\n        100625.6974,\n        101878.0284,\n        103141.6278,\n        104409.4588,\n        105686.2882,\n        106967.5402,\n        108261.6032,\n        109548.1578,\n        110852.0728,\n        112162.231,\n        113479.0072,\n        114806.2626,\n        116137.9072,\n        117469.5048,\n        118813.5186,\n        120165.4876,\n        121516.2556,\n        122875.766,\n        124250.5444,\n        125621.2222,\n        127003.2352,\n        128387.848,\n        129775.2644,\n        131181.7776,\n        132577.3086,\n        133979.9458,\n        135394.1132,\n        136800.9078,\n        138233.217,\n        139668.5308,\n        141085.212,\n        142535.2122,\n        143969.0684,\n        145420.2872,\n        146878.1542,\n        148332.7572,\n        149800.3202,\n        151269.66,\n        152743.6104,\n        154213.0948,\n        155690.288,\n        157169.4246,\n        158672.1756,\n        160160.059,\n        161650.6854,\n        163145.7772,\n        164645.6726,\n        166159.1952,\n        167682.1578,\n        169177.3328,\n        170700.0118,\n        172228.8964,\n        173732.6664,\n        175265.5556,\n        176787.799,\n        178317.111,\n        179856.6914,\n        181400.865,\n        182943.4612,\n        184486.742,\n        186033.4698,\n        187583.7886,\n        189148.1868,\n        190688.4526,\n        192250.1926,\n        193810.9042,\n        195354.2972,\n        196938.7682,\n        198493.5898,\n        200079.2824,\n        201618.912,\n        203205.5492,\n        204765.5798,\n        206356.1124,\n        207929.3064,\n        209498.7196,\n        211086.229,\n        212675.1324,\n        214256.7892,\n        215826.2392,\n        217412.8474,\n        218995.6724,\n        220618.6038,\n        222207.1166,\n        223781.0364,\n        225387.4332,\n        227005.7928,\n        228590.4336,\n        230217.8738,\n        231805.1054,\n        233408.9,\n        234995.3432,\n        236601.4956,\n        238190.7904,\n        239817.2548,\n        241411.2832,\n        243002.4066,\n        244640.1884,\n        246255.3128,\n        247849.3508,\n        249479.9734,\n        251106.8822,\n        252705.027,\n        254332.9242,\n        255935.129,\n        257526.9014,\n        259154.772,\n        260777.625,\n        262390.253,\n        264004.4906,\n        265643.59,\n        267255.4076,\n        268873.426,\n        270470.7252,\n        272106.4804,\n        273722.4456,\n        275337.794,\n        276945.7038,\n        278592.9154,\n        280204.3726,\n        281841.1606,\n        283489.171,\n        285130.1716,\n        286735.3362,\n        288364.7164,\n        289961.1814,\n        291595.5524,\n        293285.683,\n        294899.6668,\n        296499.3434,\n        298128.0462,\n        299761.8946,\n        301394.2424,\n        302997.6748,\n        304615.1478,\n        306269.7724,\n        307886.114,\n        309543.1028,\n        311153.2862,\n        312782.8546,\n        314421.2008,\n        316033.2438,\n        317692.9636,\n        319305.2648,\n        320948.7406,\n        322566.3364,\n        324228.4224,\n        325847.1542,\n    ],\n    // precision 17\n    &[\n        94542.,\n        96125.811,\n        97728.019,\n        99348.558,\n        100987.9705,\n        102646.7565,\n        104324.5125,\n        106021.7435,\n        107736.7865,\n        109469.272,\n        111223.9465,\n        112995.219,\n        114787.432,\n        116593.152,\n        118422.71,\n        120267.2345,\n        122134.6765,\n        124020.937,\n        125927.2705,\n        127851.255,\n        129788.9485,\n        131751.016,\n        133726.8225,\n        135722.592,\n        137736.789,\n        139770.568,\n        141821.518,\n        143891.343,\n        145982.1415,\n        148095.387,\n        150207.526,\n        152355.649,\n        154515.6415,\n        156696.05,\n        158887.7575,\n        161098.159,\n        163329.852,\n        165569.053,\n        167837.4005,\n        170121.6165,\n        172420.4595,\n        174732.6265,\n        177062.77,\n        179412.502,\n        181774.035,\n        184151.939,\n        186551.6895,\n        188965.691,\n        191402.8095,\n        193857.949,\n        196305.0775,\n        198774.6715,\n        201271.2585,\n        203764.78,\n        206299.3695,\n        208818.1365,\n        211373.115,\n        213946.7465,\n        216532.076,\n        219105.541,\n        221714.5375,\n        224337.5135,\n        226977.5125,\n        229613.0655,\n        232270.2685,\n        234952.2065,\n        237645.3555,\n        240331.1925,\n        243034.517,\n        245756.0725,\n        248517.6865,\n        251232.737,\n        254011.3955,\n        256785.995,\n        259556.44,\n        262368.335,\n        265156.911,\n        267965.266,\n        270785.583,\n        273616.0495,\n        276487.4835,\n        279346.639,\n        282202.509,\n        285074.3885,\n        287942.2855,\n        290856.018,\n        293774.0345,\n        296678.5145,\n        299603.6355,\n        302552.6575,\n        305492.9785,\n        308466.8605,\n        311392.581,\n        314347.538,\n        317319.4295,\n        320285.9785,\n        323301.7325,\n        326298.3235,\n        329301.3105,\n        332301.987,\n        335309.791,\n        338370.762,\n        341382.923,\n        344431.1265,\n        347464.1545,\n        350507.28,\n        353619.2345,\n        356631.2005,\n        359685.203,\n        362776.7845,\n        365886.488,\n        368958.2255,\n        372060.6825,\n        375165.4335,\n        378237.935,\n        381328.311,\n        384430.5225,\n        387576.425,\n        390683.242,\n        393839.648,\n        396977.8425,\n        400101.9805,\n        403271.296,\n        406409.8425,\n        409529.5485,\n        412678.7,\n        415847.423,\n        419020.8035,\n        422157.081,\n        425337.749,\n        428479.6165,\n        431700.902,\n        434893.1915,\n        438049.582,\n        441210.5415,\n        444379.2545,\n        447577.356,\n        450741.931,\n        453959.548,\n        457137.0935,\n        460329.846,\n        463537.4815,\n        466732.3345,\n        469960.5615,\n        473164.681,\n        476347.6345,\n        479496.173,\n        482813.1645,\n        486025.6995,\n        489249.4885,\n        492460.1945,\n        495675.8805,\n        498908.0075,\n        502131.802,\n        505374.3855,\n        508550.9915,\n        511806.7305,\n        515026.776,\n        518217.0005,\n        521523.9855,\n        524705.9855,\n        527950.997,\n        531210.0265,\n        534472.497,\n        537750.7315,\n        540926.922,\n        544207.094,\n        547429.4345,\n        550666.3745,\n        553975.3475,\n        557150.7185,\n        560399.6165,\n        563662.697,\n        566916.7395,\n        570146.1215,\n        573447.425,\n        576689.6245,\n        579874.5745,\n        583202.337,\n        586503.0255,\n        589715.635,\n        592910.161,\n        596214.3885,\n        599488.035,\n        602740.92,\n        605983.0685,\n        609248.67,\n        612491.3605,\n        615787.912,\n        619107.5245,\n        622307.9555,\n        625577.333,\n        628840.4385,\n        632085.2155,\n        635317.6135,\n        638691.7195,\n        641887.467,\n        645139.9405,\n        648441.546,\n        651666.252,\n        654941.845,\n    ],\n    // precision 18\n    &[\n        189084.,\n        192250.913,\n        195456.774,\n        198696.946,\n        201977.762,\n        205294.444,\n        208651.754,\n        212042.099,\n        215472.269,\n        218941.91,\n        222443.912,\n        225996.845,\n        229568.199,\n        233193.568,\n        236844.457,\n        240543.233,\n        244279.475,\n        248044.27,\n        251854.588,\n        255693.2,\n        259583.619,\n        263494.621,\n        267445.385,\n        271454.061,\n        275468.769,\n        279549.456,\n        283646.446,\n        287788.198,\n        291966.099,\n        296181.164,\n        300431.469,\n        304718.618,\n        309024.004,\n        313393.508,\n        317760.803,\n        322209.731,\n        326675.061,\n        331160.627,\n        335654.47,\n        340241.442,\n        344841.833,\n        349467.132,\n        354130.629,\n        358819.432,\n        363574.626,\n        368296.587,\n        373118.482,\n        377914.93,\n        382782.301,\n        387680.669,\n        392601.981,\n        397544.323,\n        402529.115,\n        407546.018,\n        412593.658,\n        417638.657,\n        422762.865,\n        427886.169,\n        433017.167,\n        438213.273,\n        443441.254,\n        448692.421,\n        453937.533,\n        459239.049,\n        464529.569,\n        469910.083,\n        475274.03,\n        480684.473,\n        486070.26,\n        491515.237,\n        496995.651,\n        502476.617,\n        507973.609,\n        513497.19,\n        519083.233,\n        524726.509,\n        530305.505,\n        535945.728,\n        541584.404,\n        547274.055,\n        552967.236,\n        558667.862,\n        564360.216,\n        570128.148,\n        575965.08,\n        581701.952,\n        587532.523,\n        593361.144,\n        599246.128,\n        605033.418,\n        610958.779,\n        616837.117,\n        622772.818,\n        628672.04,\n        634675.369,\n        640574.831,\n        646585.739,\n        652574.547,\n        658611.217,\n        664642.684,\n        670713.914,\n        676737.681,\n        682797.313,\n        688837.897,\n        694917.874,\n        701009.882,\n        707173.648,\n        713257.254,\n        719415.392,\n        725636.761,\n        731710.697,\n        737906.209,\n        744103.074,\n        750313.39,\n        756504.185,\n        762712.579,\n        768876.985,\n        775167.859,\n        781359.,\n        787615.959,\n        793863.597,\n        800245.477,\n        806464.582,\n        812785.294,\n        819005.925,\n        825403.057,\n        831676.197,\n        837936.284,\n        844266.968,\n        850642.711,\n        856959.756,\n        863322.774,\n        869699.931,\n        876102.478,\n        882355.787,\n        888694.463,\n        895159.952,\n        901536.143,\n        907872.631,\n        914293.672,\n        920615.14,\n        927130.974,\n        933409.404,\n        939922.178,\n        946331.47,\n        952745.93,\n        959209.264,\n        965590.224,\n        972077.284,\n        978501.961,\n        984953.19,\n        991413.271,\n        997817.479,\n        1004222.658,\n        1010725.676,\n        1017177.138,\n        1023612.529,\n        1030098.236,\n        1036493.719,\n        1043112.207,\n        1049537.036,\n        1056008.096,\n        1062476.184,\n        1068942.337,\n        1075524.95,\n        1081932.864,\n        1088426.025,\n        1094776.005,\n        1101327.448,\n        1107901.673,\n        1114423.639,\n        1120884.602,\n        1127324.923,\n        1133794.24,\n        1140328.886,\n        1146849.376,\n        1153346.682,\n        1159836.502,\n        1166478.703,\n        1172953.304,\n        1179391.502,\n        1185950.982,\n        1192544.052,\n        1198913.41,\n        1205430.994,\n        1212015.525,\n        1218674.042,\n        1225121.683,\n        1231551.101,\n        1238126.379,\n        1244673.795,\n        1251260.649,\n        1257697.86,\n        1264320.983,\n        1270736.319,\n        1277274.694,\n        1283804.95,\n        1290211.514,\n        1296858.568,\n        1303455.691,\n    ],\n];\n\npub(crate) const BIAS_DATA_OFFSET: usize = 4;\npub(crate) const BIAS_DATA_VEC: &[&[f64]] = &[\n    // precision 4\n    &[\n        10.,\n        9.717,\n        9.207,\n        8.7896,\n        8.2882,\n        7.8204,\n        7.3772,\n        6.9342,\n        6.5202,\n        6.161,\n        5.7722,\n        5.4636,\n        5.0396,\n        4.6766,\n        4.3566,\n        4.0454,\n        3.7936,\n        3.4856,\n        3.2666,\n        2.9946,\n        2.766,\n        2.4692,\n        2.3638,\n        2.0764,\n        1.7864,\n        1.7602,\n        1.4814,\n        1.433,\n        1.2926,\n        1.0664,\n        0.999600000000001,\n        0.7956,\n        0.5366,\n        0.589399999999998,\n        0.573799999999999,\n        0.269799999999996,\n        0.368200000000002,\n        0.0544000000000011,\n        0.234200000000001,\n        0.0108000000000033,\n        -0.203400000000002,\n        -0.0701999999999998,\n        -0.129600000000003,\n        -0.364199999999997,\n        -0.480600000000003,\n        -0.226999999999997,\n        -0.322800000000001,\n        -0.382599999999996,\n        -0.511200000000002,\n        -0.669600000000003,\n        -0.749400000000001,\n        -0.500399999999999,\n        -0.617600000000003,\n        -0.6922,\n        -0.601599999999998,\n        -0.416200000000003,\n        -0.338200000000001,\n        -0.782600000000002,\n        -0.648600000000002,\n        -0.919800000000002,\n        -0.851799999999997,\n        -0.962400000000002,\n        -0.6402,\n        -1.1922,\n        -1.0256,\n        -1.086,\n        -1.21899999999999,\n        -0.819400000000002,\n        -0.940600000000003,\n        -1.1554,\n        -1.2072,\n        -1.1752,\n        -1.16759999999999,\n        -1.14019999999999,\n        -1.3754,\n        -1.29859999999999,\n        -1.607,\n        -1.3292,\n        -1.7606,\n    ],\n    // precision 5\n    &[\n        22.,\n        21.1194,\n        20.8208,\n        20.2318,\n        19.77,\n        19.2436,\n        18.7774,\n        18.2848,\n        17.8224,\n        17.3742,\n        16.9336,\n        16.503,\n        16.0494,\n        15.6292,\n        15.2124,\n        14.798,\n        14.367,\n        13.9728,\n        13.5944,\n        13.217,\n        12.8438,\n        12.3696,\n        12.0956,\n        11.7044,\n        11.324,\n        11.0668,\n        10.6698,\n        10.3644,\n        10.049,\n        9.6918,\n        9.4146,\n        9.082,\n        8.687,\n        8.5398,\n        8.2462,\n        7.857,\n        7.6606,\n        7.4168,\n        7.1248,\n        6.9222,\n        6.6804,\n        6.447,\n        6.3454,\n        5.9594,\n        5.7636,\n        5.5776,\n        5.331,\n        5.19,\n        4.9676,\n        4.7564,\n        4.5314,\n        4.4442,\n        4.3708,\n        3.9774,\n        3.9624,\n        3.8796,\n        3.755,\n        3.472,\n        3.2076,\n        3.1024,\n        2.8908,\n        2.7338,\n        2.7728,\n        2.629,\n        2.413,\n        2.3266,\n        2.1524,\n        2.2642,\n        2.1806,\n        2.0566,\n        1.9192,\n        1.7598,\n        1.3516,\n        1.5802,\n        1.43859999999999,\n        1.49160000000001,\n        1.1524,\n        1.1892,\n        0.841399999999993,\n        0.879800000000003,\n        0.837599999999995,\n        0.469800000000006,\n        0.765600000000006,\n        0.331000000000003,\n        0.591399999999993,\n        0.601200000000006,\n        0.701599999999999,\n        0.558199999999999,\n        0.339399999999998,\n        0.354399999999998,\n        0.491200000000006,\n        0.308000000000007,\n        0.355199999999996,\n        -0.0254000000000048,\n        0.205200000000005,\n        -0.272999999999996,\n        0.132199999999997,\n        0.394400000000005,\n        -0.241200000000006,\n        0.242000000000004,\n        0.191400000000002,\n        0.253799999999998,\n        -0.122399999999999,\n        -0.370800000000003,\n        0.193200000000004,\n        -0.0848000000000013,\n        0.0867999999999967,\n        -0.327200000000005,\n        -0.285600000000002,\n        0.311400000000006,\n        -0.128399999999999,\n        -0.754999999999995,\n        -0.209199999999996,\n        -0.293599999999998,\n        -0.364000000000004,\n        -0.253600000000006,\n        -0.821200000000005,\n        -0.253600000000006,\n        -0.510400000000004,\n        -0.383399999999995,\n        -0.491799999999998,\n        -0.220200000000006,\n        -0.0972000000000008,\n        -0.557400000000001,\n        -0.114599999999996,\n        -0.295000000000002,\n        -0.534800000000004,\n        0.346399999999988,\n        -0.65379999999999,\n        0.0398000000000138,\n        0.0341999999999985,\n        -0.995800000000003,\n        -0.523400000000009,\n        -0.489000000000004,\n        -0.274799999999999,\n        -0.574999999999989,\n        -0.482799999999997,\n        0.0571999999999946,\n        -0.330600000000004,\n        -0.628800000000012,\n        -0.140199999999993,\n        -0.540600000000012,\n        -0.445999999999998,\n        -0.599400000000003,\n        -0.262599999999992,\n        0.163399999999996,\n        -0.100599999999986,\n        -0.39500000000001,\n        -1.06960000000001,\n        -0.836399999999998,\n        -0.753199999999993,\n        -0.412399999999991,\n        -0.790400000000005,\n        -0.29679999999999,\n        -0.28540000000001,\n        -0.193000000000012,\n        -0.0772000000000048,\n        -0.962799999999987,\n        -0.414800000000014,\n    ],\n    // precision 6\n    &[\n        45.,\n        44.1902,\n        43.271,\n        42.8358,\n        41.8142,\n        41.2854,\n        40.317,\n        39.354,\n        38.8924,\n        37.9436,\n        37.4596,\n        36.5262,\n        35.6248,\n        35.1574,\n        34.2822,\n        33.837,\n        32.9636,\n        32.074,\n        31.7042,\n        30.7976,\n        30.4772,\n        29.6564,\n        28.7942,\n        28.5004,\n        27.686,\n        27.291,\n        26.5672,\n        25.8556,\n        25.4982,\n        24.8204,\n        24.4252,\n        23.7744,\n        23.0786,\n        22.8344,\n        22.0294,\n        21.8098,\n        21.0794,\n        20.5732,\n        20.1878,\n        19.5648,\n        19.2902,\n        18.6784,\n        18.3352,\n        17.8946,\n        17.3712,\n        17.0852,\n        16.499,\n        16.2686,\n        15.6844,\n        15.2234,\n        14.9732,\n        14.3356,\n        14.2286,\n        13.7262,\n        13.3284,\n        13.1048,\n        12.5962,\n        12.3562,\n        12.1272,\n        11.4184,\n        11.4974,\n        11.0822,\n        10.856,\n        10.48,\n        10.2834,\n        10.0208,\n        9.637,\n        9.51739999999999,\n        9.05759999999999,\n        8.74760000000001,\n        8.42700000000001,\n        8.1326,\n        8.2372,\n        8.2788,\n        7.6776,\n        7.79259999999999,\n        7.1952,\n        6.9564,\n        6.6454,\n        6.87,\n        6.5428,\n        6.19999999999999,\n        6.02940000000001,\n        5.62780000000001,\n        5.6782,\n        5.792,\n        5.35159999999999,\n        5.28319999999999,\n        5.0394,\n        5.07480000000001,\n        4.49119999999999,\n        4.84899999999999,\n        4.696,\n        4.54040000000001,\n        4.07300000000001,\n        4.37139999999999,\n        3.7216,\n        3.7328,\n        3.42080000000001,\n        3.41839999999999,\n        3.94239999999999,\n        3.27719999999999,\n        3.411,\n        3.13079999999999,\n        2.76900000000001,\n        2.92580000000001,\n        2.68279999999999,\n        2.75020000000001,\n        2.70599999999999,\n        2.3886,\n        3.01859999999999,\n        2.45179999999999,\n        2.92699999999999,\n        2.41720000000001,\n        2.41139999999999,\n        2.03299999999999,\n        2.51240000000001,\n        2.5564,\n        2.60079999999999,\n        2.41720000000001,\n        1.80439999999999,\n        1.99700000000001,\n        2.45480000000001,\n        1.8948,\n        2.2346,\n        2.30860000000001,\n        2.15479999999999,\n        1.88419999999999,\n        1.6508,\n        0.677199999999999,\n        1.72540000000001,\n        1.4752,\n        1.72280000000001,\n        1.66139999999999,\n        1.16759999999999,\n        1.79300000000001,\n        1.00059999999999,\n        0.905200000000008,\n        0.659999999999997,\n        1.55879999999999,\n        1.1636,\n        0.688199999999995,\n        0.712600000000009,\n        0.450199999999995,\n        1.1978,\n        0.975599999999986,\n        0.165400000000005,\n        1.727,\n        1.19739999999999,\n        -0.252600000000001,\n        1.13460000000001,\n        1.3048,\n        1.19479999999999,\n        0.313400000000001,\n        0.878999999999991,\n        1.12039999999999,\n        0.853000000000009,\n        1.67920000000001,\n        0.856999999999999,\n        0.448599999999999,\n        1.2362,\n        0.953399999999988,\n        1.02859999999998,\n        0.563199999999995,\n        0.663000000000011,\n        0.723000000000013,\n        0.756599999999992,\n        0.256599999999992,\n        -0.837600000000009,\n        0.620000000000005,\n        0.821599999999989,\n        0.216600000000028,\n        0.205600000000004,\n        0.220199999999977,\n        0.372599999999977,\n        0.334400000000016,\n        0.928400000000011,\n        0.972800000000007,\n        0.192400000000021,\n        0.487199999999973,\n        -0.413000000000011,\n        0.807000000000016,\n        0.120600000000024,\n        0.769000000000005,\n        0.870799999999974,\n        0.66500000000002,\n        0.118200000000002,\n        0.401200000000017,\n        0.635199999999998,\n        0.135400000000004,\n        0.175599999999974,\n        1.16059999999999,\n        0.34620000000001,\n        0.521400000000028,\n        -0.586599999999976,\n        -1.16480000000001,\n        0.968399999999974,\n        0.836999999999989,\n        0.779600000000016,\n        0.985799999999983,\n    ],\n    // precision 7\n    &[\n        91.,\n        89.4934,\n        87.9758,\n        86.4574,\n        84.9718,\n        83.4954,\n        81.5302,\n        80.0756,\n        78.6374,\n        77.1782,\n        75.7888,\n        73.9522,\n        72.592,\n        71.2532,\n        69.9086,\n        68.5938,\n        66.9474,\n        65.6796,\n        64.4394,\n        63.2176,\n        61.9768,\n        60.4214,\n        59.2528,\n        58.0102,\n        56.8658,\n        55.7278,\n        54.3044,\n        53.1316,\n        52.093,\n        51.0032,\n        49.9092,\n        48.6306,\n        47.5294,\n        46.5756,\n        45.6508,\n        44.662,\n        43.552,\n        42.3724,\n        41.617,\n        40.5754,\n        39.7872,\n        38.8444,\n        37.7988,\n        36.8606,\n        36.2118,\n        35.3566,\n        34.4476,\n        33.5882,\n        32.6816,\n        32.0824,\n        31.0258,\n        30.6048,\n        29.4436,\n        28.7274,\n        27.957,\n        27.147,\n        26.4364,\n        25.7592,\n        25.3386,\n        24.781,\n        23.8028,\n        23.656,\n        22.6544,\n        21.996,\n        21.4718,\n        21.1544,\n        20.6098,\n        19.5956,\n        19.0616,\n        18.5758,\n        18.4878,\n        17.5244,\n        17.2146,\n        16.724,\n        15.8722,\n        15.5198,\n        15.0414,\n        14.941,\n        14.9048,\n        13.87,\n        13.4304,\n        13.028,\n        12.4708,\n        12.37,\n        12.0624,\n        11.4668,\n        11.5532,\n        11.4352,\n        11.2564,\n        10.2744,\n        10.2118,\n        9.74720000000002,\n        10.1456,\n        9.2928,\n        8.75040000000001,\n        8.55279999999999,\n        8.97899999999998,\n        8.21019999999999,\n        8.18340000000001,\n        7.3494,\n        7.32499999999999,\n        7.66140000000001,\n        6.90300000000002,\n        7.25439999999998,\n        6.9042,\n        7.21499999999997,\n        6.28640000000001,\n        6.08139999999997,\n        6.6764,\n        6.30099999999999,\n        5.13900000000001,\n        5.65800000000002,\n        5.17320000000001,\n        4.59019999999998,\n        4.9538,\n        5.08280000000002,\n        4.92200000000003,\n        4.99020000000002,\n        4.7328,\n        5.4538,\n        4.11360000000002,\n        4.22340000000003,\n        4.08780000000002,\n        3.70800000000003,\n        4.15559999999999,\n        4.18520000000001,\n        3.63720000000001,\n        3.68220000000002,\n        3.77960000000002,\n        3.6078,\n        2.49160000000001,\n        3.13099999999997,\n        2.5376,\n        3.19880000000001,\n        3.21100000000001,\n        2.4502,\n        3.52820000000003,\n        2.91199999999998,\n        3.04480000000001,\n        2.7432,\n        2.85239999999999,\n        2.79880000000003,\n        2.78579999999999,\n        1.88679999999999,\n        2.98860000000002,\n        2.50639999999999,\n        1.91239999999999,\n        2.66160000000002,\n        2.46820000000002,\n        1.58199999999999,\n        1.30399999999997,\n        2.27379999999999,\n        2.68939999999998,\n        1.32900000000001,\n        3.10599999999999,\n        1.69080000000002,\n        2.13740000000001,\n        2.53219999999999,\n        1.88479999999998,\n        1.33240000000001,\n        1.45119999999997,\n        1.17899999999997,\n        2.44119999999998,\n        1.60659999999996,\n        2.16700000000003,\n        0.77940000000001,\n        2.37900000000002,\n        2.06700000000001,\n        1.46000000000004,\n        2.91160000000002,\n        1.69200000000001,\n        0.954600000000028,\n        2.49300000000005,\n        2.2722,\n        1.33500000000004,\n        2.44899999999996,\n        1.20140000000004,\n        3.07380000000001,\n        2.09739999999999,\n        2.85640000000001,\n        2.29960000000005,\n        2.40899999999999,\n        1.97040000000004,\n        0.809799999999996,\n        1.65279999999996,\n        2.59979999999996,\n        0.95799999999997,\n        2.06799999999998,\n        2.32780000000002,\n        4.20159999999998,\n        1.96320000000003,\n        1.86400000000003,\n        1.42999999999995,\n        3.77940000000001,\n        1.27200000000005,\n        1.86440000000005,\n        2.20600000000002,\n        3.21900000000005,\n        1.5154,\n        2.61019999999996,\n    ],\n    // precision 8\n    &[\n        183.2152,\n        180.2454,\n        177.2096,\n        173.6652,\n        170.6312,\n        167.6822,\n        164.249,\n        161.3296,\n        158.0038,\n        155.2074,\n        152.4612,\n        149.27,\n        146.5178,\n        143.4412,\n        140.8032,\n        138.1634,\n        135.1688,\n        132.6074,\n        129.6946,\n        127.2664,\n        124.8228,\n        122.0432,\n        119.6824,\n        116.9464,\n        114.6268,\n        112.2626,\n        109.8376,\n        107.4034,\n        104.8956,\n        102.8522,\n        100.7638,\n        98.3552,\n        96.3556,\n        93.7526,\n        91.9292,\n        89.8954,\n        87.8198,\n        85.7668,\n        83.298,\n        81.6688,\n        79.9466,\n        77.9746,\n        76.1672,\n        74.3474,\n        72.3028,\n        70.8912,\n        69.114,\n        67.4646,\n        65.9744,\n        64.4092,\n        62.6022,\n        60.843,\n        59.5684,\n        58.1652,\n        56.5426,\n        55.4152,\n        53.5388,\n        52.3592,\n        51.1366,\n        49.486,\n        48.3918,\n        46.5076,\n        45.509,\n        44.3834,\n        43.3498,\n        42.0668,\n        40.7346,\n        40.1228,\n        38.4528,\n        37.7,\n        36.644,\n        36.0518,\n        34.5774,\n        33.9068,\n        32.432,\n        32.1666,\n        30.434,\n        29.6644,\n        28.4894,\n        27.6312,\n        26.3804,\n        26.292,\n        25.5496000000001,\n        25.0234,\n        24.8206,\n        22.6146,\n        22.4188,\n        22.117,\n        20.6762,\n        20.6576,\n        19.7864,\n        19.509,\n        18.5334,\n        17.9204,\n        17.772,\n        16.2924,\n        16.8654,\n        15.1836,\n        15.745,\n        15.1316,\n        15.0386,\n        14.0136,\n        13.6342,\n        12.6196,\n        12.1866,\n        12.4281999999999,\n        11.3324,\n        10.4794000000001,\n        11.5038,\n        10.129,\n        9.52800000000002,\n        10.3203999999999,\n        9.46299999999997,\n        9.79280000000006,\n        9.12300000000005,\n        8.74180000000001,\n        9.2192,\n        7.51020000000005,\n        7.60659999999996,\n        7.01840000000004,\n        7.22239999999999,\n        7.40139999999997,\n        6.76179999999999,\n        7.14359999999999,\n        5.65060000000005,\n        5.63779999999997,\n        5.76599999999996,\n        6.75139999999999,\n        5.57759999999996,\n        3.73220000000003,\n        5.8048,\n        5.63019999999995,\n        4.93359999999996,\n        3.47979999999995,\n        4.33879999999999,\n        3.98940000000005,\n        3.81960000000004,\n        3.31359999999995,\n        3.23080000000004,\n        3.4588,\n        3.08159999999998,\n        3.4076,\n        3.00639999999999,\n        2.38779999999997,\n        2.61900000000003,\n        1.99800000000005,\n        3.34820000000002,\n        2.95060000000001,\n        0.990999999999985,\n        2.11440000000005,\n        2.20299999999997,\n        2.82219999999995,\n        2.73239999999998,\n        2.7826,\n        3.76660000000004,\n        2.26480000000004,\n        2.31280000000004,\n        2.40819999999997,\n        2.75360000000001,\n        3.33759999999995,\n        2.71559999999999,\n        1.7478000000001,\n        1.42920000000004,\n        2.39300000000003,\n        2.22779999999989,\n        2.34339999999997,\n        0.87259999999992,\n        3.88400000000001,\n        1.80600000000004,\n        1.91759999999999,\n        1.16779999999994,\n        1.50320000000011,\n        2.52500000000009,\n        0.226400000000012,\n        2.31500000000005,\n        0.930000000000064,\n        1.25199999999995,\n        2.14959999999996,\n        0.0407999999999902,\n        2.5447999999999,\n        1.32960000000003,\n        0.197400000000016,\n        2.52620000000002,\n        3.33279999999991,\n        -1.34300000000007,\n        0.422199999999975,\n        0.917200000000093,\n        1.12920000000008,\n        1.46060000000011,\n        1.45779999999991,\n        2.8728000000001,\n        3.33359999999993,\n        -1.34079999999994,\n        1.57680000000005,\n        0.363000000000056,\n        1.40740000000005,\n        0.656600000000026,\n        0.801400000000058,\n        -0.454600000000028,\n        1.51919999999996,\n    ],\n    // precision 9\n    &[\n        368.,\n        361.8294,\n        355.2452,\n        348.6698,\n        342.1464,\n        336.2024,\n        329.8782,\n        323.6598,\n        317.462,\n        311.2826,\n        305.7102,\n        299.7416,\n        293.9366,\n        288.1046,\n        282.285,\n        277.0668,\n        271.306,\n        265.8448,\n        260.301,\n        254.9886,\n        250.2422,\n        244.8138,\n        239.7074,\n        234.7428,\n        229.8402,\n        225.1664,\n        220.3534,\n        215.594,\n        210.6886,\n        205.7876,\n        201.65,\n        197.228,\n        192.8036,\n        188.1666,\n        184.0818,\n        180.0824,\n        176.2574,\n        172.302,\n        168.1644,\n        164.0056,\n        160.3802,\n        156.7192,\n        152.5234,\n        149.2084,\n        145.831,\n        142.485,\n        139.1112,\n        135.4764,\n        131.76,\n        129.3368,\n        126.5538,\n        122.5058,\n        119.2646,\n        116.5902,\n        113.3818,\n        110.8998,\n        107.9532,\n        105.2062,\n        102.2798,\n        99.4728,\n        96.9582,\n        94.3292,\n        92.171,\n        89.7809999999999,\n        87.5716,\n        84.7048,\n        82.5322,\n        79.875,\n        78.3972,\n        75.3464,\n        73.7274,\n        71.2834,\n        70.1444,\n        68.4263999999999,\n        66.0166,\n        64.018,\n        62.0437999999999,\n        60.3399999999999,\n        58.6856,\n        57.9836,\n        55.0311999999999,\n        54.6769999999999,\n        52.3188,\n        51.4846,\n        49.4423999999999,\n        47.739,\n        46.1487999999999,\n        44.9202,\n        43.4059999999999,\n        42.5342000000001,\n        41.2834,\n        38.8954000000001,\n        38.3286000000001,\n        36.2146,\n        36.6684,\n        35.9946,\n        33.123,\n        33.4338,\n        31.7378000000001,\n        29.076,\n        28.9692,\n        27.4964,\n        27.0998,\n        25.9864,\n        26.7754,\n        24.3208,\n        23.4838,\n        22.7388000000001,\n        24.0758000000001,\n        21.9097999999999,\n        20.9728,\n        19.9228000000001,\n        19.9292,\n        16.617,\n        17.05,\n        18.2996000000001,\n        15.6128000000001,\n        15.7392,\n        14.5174,\n        13.6322,\n        12.2583999999999,\n        13.3766000000001,\n        11.423,\n        13.1232,\n        9.51639999999998,\n        10.5938000000001,\n        9.59719999999993,\n        8.12220000000002,\n        9.76739999999995,\n        7.50440000000003,\n        7.56999999999994,\n        6.70440000000008,\n        6.41419999999994,\n        6.71019999999999,\n        5.60940000000005,\n        4.65219999999999,\n        6.84099999999989,\n        3.4072000000001,\n        3.97859999999991,\n        3.32760000000007,\n        5.52160000000003,\n        3.31860000000006,\n        2.06940000000009,\n        4.35400000000004,\n        1.57500000000005,\n        0.280799999999999,\n        2.12879999999996,\n        -0.214799999999968,\n        -0.0378000000000611,\n        -0.658200000000079,\n        0.654800000000023,\n        -0.0697999999999865,\n        0.858400000000074,\n        -2.52700000000004,\n        -2.1751999999999,\n        -3.35539999999992,\n        -1.04019999999991,\n        -0.651000000000067,\n        -2.14439999999991,\n        -1.96659999999997,\n        -3.97939999999994,\n        -0.604400000000169,\n        -3.08260000000018,\n        -3.39159999999993,\n        -5.29640000000018,\n        -5.38920000000007,\n        -5.08759999999984,\n        -4.69900000000007,\n        -5.23720000000003,\n        -3.15779999999995,\n        -4.97879999999986,\n        -4.89899999999989,\n        -7.48880000000008,\n        -5.94799999999987,\n        -5.68060000000014,\n        -6.67180000000008,\n        -4.70499999999993,\n        -7.27779999999984,\n        -4.6579999999999,\n        -4.4362000000001,\n        -4.32139999999981,\n        -5.18859999999995,\n        -6.66879999999992,\n        -6.48399999999992,\n        -5.1260000000002,\n        -4.4032000000002,\n        -6.13500000000022,\n        -5.80819999999994,\n        -4.16719999999987,\n        -4.15039999999999,\n        -7.45600000000013,\n        -7.24080000000004,\n        -9.83179999999993,\n        -5.80420000000004,\n        -8.6561999999999,\n        -6.99940000000015,\n        -10.5473999999999,\n        -7.34139999999979,\n        -6.80999999999995,\n        -6.29719999999998,\n        -6.23199999999997,\n    ],\n    // precision 10\n    &[\n        737.1256,\n        724.4234,\n        711.1064,\n        698.4732,\n        685.4636,\n        673.0644,\n        660.488,\n        647.9654,\n        636.0832,\n        623.7864,\n        612.1992,\n        600.2176,\n        588.5228,\n        577.1716,\n        565.7752,\n        554.899,\n        543.6126,\n        532.6492,\n        521.9474,\n        511.5214,\n        501.1064,\n        490.6364,\n        480.2468,\n        470.4588,\n        460.3832,\n        451.0584,\n        440.8606,\n        431.3868,\n        422.5062,\n        413.1862,\n        404.463,\n        395.339,\n        386.1936,\n        378.1292,\n        369.1854,\n        361.2908,\n        353.3324,\n        344.8518,\n        337.5204,\n        329.4854,\n        321.9318,\n        314.552,\n        306.4658,\n        299.4256,\n        292.849,\n        286.152,\n        278.8956,\n        271.8792,\n        265.118,\n        258.62,\n        252.5132,\n        245.9322,\n        239.7726,\n        233.6086,\n        227.5332,\n        222.5918,\n        216.4294,\n        210.7662,\n        205.4106,\n        199.7338,\n        194.9012,\n        188.4486,\n        183.1556,\n        178.6338,\n        173.7312,\n        169.6264,\n        163.9526,\n        159.8742,\n        155.8326,\n        151.1966,\n        147.5594,\n        143.07,\n        140.037,\n        134.1804,\n        131.071,\n        127.4884,\n        124.0848,\n        120.2944,\n        117.333,\n        112.9626,\n        110.2902,\n        107.0814,\n        103.0334,\n        99.4832000000001,\n        96.3899999999999,\n        93.7202000000002,\n        90.1714000000002,\n        87.2357999999999,\n        85.9346,\n        82.8910000000001,\n        80.0264000000002,\n        78.3834000000002,\n        75.1543999999999,\n        73.8683999999998,\n        70.9895999999999,\n        69.4367999999999,\n        64.8701999999998,\n        65.0408000000002,\n        61.6738,\n        59.5207999999998,\n        57.0158000000001,\n        54.2302,\n        53.0962,\n        50.4985999999999,\n        52.2588000000001,\n        47.3914,\n        45.6244000000002,\n        42.8377999999998,\n        43.0072,\n        40.6516000000001,\n        40.2453999999998,\n        35.2136,\n        36.4546,\n        33.7849999999999,\n        33.2294000000002,\n        32.4679999999998,\n        30.8670000000002,\n        28.6507999999999,\n        28.9099999999999,\n        27.5983999999999,\n        26.1619999999998,\n        24.5563999999999,\n        23.2328000000002,\n        21.9484000000002,\n        21.5902000000001,\n        21.3346000000001,\n        17.7031999999999,\n        20.6111999999998,\n        19.5545999999999,\n        15.7375999999999,\n        17.0720000000001,\n        16.9517999999998,\n        15.326,\n        13.1817999999998,\n        14.6925999999999,\n        13.0859999999998,\n        13.2754,\n        10.8697999999999,\n        11.248,\n        7.3768,\n        4.72339999999986,\n        7.97899999999981,\n        8.7503999999999,\n        7.68119999999999,\n        9.7199999999998,\n        7.73919999999998,\n        5.6224000000002,\n        7.44560000000001,\n        6.6601999999998,\n        5.9058,\n        4.00199999999995,\n        4.51699999999983,\n        4.68240000000014,\n        3.86220000000003,\n        5.13639999999987,\n        5.98500000000013,\n        2.47719999999981,\n        2.61999999999989,\n        1.62800000000016,\n        4.65000000000009,\n        0.225599999999758,\n        0.831000000000131,\n        -0.359400000000278,\n        1.27599999999984,\n        -2.92559999999958,\n        -0.0303999999996449,\n        2.37079999999969,\n        -2.0033999999996,\n        0.804600000000391,\n        0.30199999999968,\n        1.1247999999996,\n        -2.6880000000001,\n        0.0321999999996478,\n        -1.18099999999959,\n        -3.9402,\n        -1.47940000000017,\n        -0.188400000000001,\n        -2.10720000000038,\n        -2.04159999999956,\n        -3.12880000000041,\n        -4.16160000000036,\n        -0.612799999999879,\n        -3.48719999999958,\n        -8.17900000000009,\n        -5.37780000000021,\n        -4.01379999999972,\n        -5.58259999999973,\n        -5.73719999999958,\n        -7.66799999999967,\n        -5.69520000000011,\n        -1.1247999999996,\n        -5.58520000000044,\n        -8.04560000000038,\n        -4.64840000000004,\n        -11.6468000000004,\n        -7.97519999999986,\n        -5.78300000000036,\n        -7.67420000000038,\n        -10.6328000000003,\n        -9.81720000000041,\n    ],\n    // precision 11\n    &[\n        1476.,\n        1449.6014,\n        1423.5802,\n        1397.7942,\n        1372.3042,\n        1347.2062,\n        1321.8402,\n        1297.2292,\n        1272.9462,\n        1248.9926,\n        1225.3026,\n        1201.4252,\n        1178.0578,\n        1155.6092,\n        1132.626,\n        1110.5568,\n        1088.527,\n        1066.5154,\n        1045.1874,\n        1024.3878,\n        1003.37,\n        982.1972,\n        962.5728,\n        942.1012,\n        922.9668,\n        903.292,\n        884.0772,\n        864.8578,\n        846.6562,\n        828.041,\n        809.714,\n        792.3112,\n        775.1806,\n        757.9854,\n        740.656,\n        724.346,\n        707.5154,\n        691.8378,\n        675.7448,\n        659.6722,\n        645.5722,\n        630.1462,\n        614.4124,\n        600.8728,\n        585.898,\n        572.408,\n        558.4926,\n        544.4938,\n        531.6776,\n        517.282,\n        505.7704,\n        493.1012,\n        480.7388,\n        467.6876,\n        456.1872,\n        445.5048,\n        433.0214,\n        420.806,\n        411.409,\n        400.4144,\n        389.4294,\n        379.2286,\n        369.651,\n        360.6156,\n        350.337,\n        342.083,\n        332.1538,\n        322.5094,\n        315.01,\n        305.6686,\n        298.1678,\n        287.8116,\n        280.9978,\n        271.9204,\n        265.3286,\n        257.5706,\n        249.6014,\n        242.544,\n        235.5976,\n        229.583,\n        220.9438,\n        214.672,\n        208.2786,\n        201.8628,\n        195.1834,\n        191.505,\n        186.1816,\n        178.5188,\n        172.2294,\n        167.8908,\n        161.0194,\n        158.052,\n        151.4588,\n        148.1596,\n        143.4344,\n        138.5238,\n        133.13,\n        127.6374,\n        124.8162,\n        118.7894,\n        117.3984,\n        114.6078,\n        109.0858,\n        105.1036,\n        103.6258,\n        98.6018000000004,\n        95.7618000000002,\n        93.5821999999998,\n        88.5900000000001,\n        86.9992000000002,\n        82.8800000000001,\n        80.4539999999997,\n        74.6981999999998,\n        74.3644000000004,\n        73.2914000000001,\n        65.5709999999999,\n        66.9232000000002,\n        65.1913999999997,\n        62.5882000000001,\n        61.5702000000001,\n        55.7035999999998,\n        56.1764000000003,\n        52.7596000000003,\n        53.0302000000001,\n        49.0609999999997,\n        48.4694,\n        44.933,\n        46.0474000000004,\n        44.7165999999997,\n        41.9416000000001,\n        39.9207999999999,\n        35.6328000000003,\n        35.5276000000003,\n        33.1934000000001,\n        33.2371999999996,\n        33.3864000000003,\n        33.9228000000003,\n        30.2371999999996,\n        29.1373999999996,\n        25.2272000000003,\n        24.2942000000003,\n        19.8338000000003,\n        18.9005999999999,\n        23.0907999999999,\n        21.8544000000002,\n        19.5176000000001,\n        15.4147999999996,\n        16.9314000000004,\n        18.6737999999996,\n        12.9877999999999,\n        14.3688000000002,\n        12.0447999999997,\n        15.5219999999999,\n        12.5299999999997,\n        14.5940000000001,\n        14.3131999999996,\n        9.45499999999993,\n        12.9441999999999,\n        3.91139999999996,\n        13.1373999999996,\n        5.44720000000052,\n        9.82779999999912,\n        7.87279999999919,\n        3.67760000000089,\n        5.46980000000076,\n        5.55099999999948,\n        5.65979999999945,\n        3.89439999999922,\n        3.1275999999998,\n        5.65140000000065,\n        6.3062000000009,\n        3.90799999999945,\n        1.87060000000019,\n        5.17020000000048,\n        2.46680000000015,\n        0.770000000000437,\n        -3.72340000000077,\n        1.16400000000067,\n        8.05340000000069,\n        0.135399999999208,\n        2.15940000000046,\n        0.766999999999825,\n        1.0594000000001,\n        3.15500000000065,\n        -0.287399999999252,\n        2.37219999999979,\n        -2.86620000000039,\n        -1.63199999999961,\n        -2.22979999999916,\n        -0.15519999999924,\n        -1.46039999999994,\n        -0.262199999999211,\n        -2.34460000000036,\n        -2.8078000000005,\n        -3.22179999999935,\n        -5.60159999999996,\n        -8.42200000000048,\n        -9.43740000000071,\n        0.161799999999857,\n        -10.4755999999998,\n        -10.0823999999993,\n    ],\n    // precision 12\n    &[\n        2953.,\n        2900.4782,\n        2848.3568,\n        2796.3666,\n        2745.324,\n        2694.9598,\n        2644.648,\n        2595.539,\n        2546.1474,\n        2498.2576,\n        2450.8376,\n        2403.6076,\n        2357.451,\n        2311.38,\n        2266.4104,\n        2221.5638,\n        2176.9676,\n        2134.193,\n        2090.838,\n        2048.8548,\n        2007.018,\n        1966.1742,\n        1925.4482,\n        1885.1294,\n        1846.4776,\n        1807.4044,\n        1768.8724,\n        1731.3732,\n        1693.4304,\n        1657.5326,\n        1621.949,\n        1586.5532,\n        1551.7256,\n        1517.6182,\n        1483.5186,\n        1450.4528,\n        1417.865,\n        1385.7164,\n        1352.6828,\n        1322.6708,\n        1291.8312,\n        1260.9036,\n        1231.476,\n        1201.8652,\n        1173.6718,\n        1145.757,\n        1119.2072,\n        1092.2828,\n        1065.0434,\n        1038.6264,\n        1014.3192,\n        988.5746,\n        965.0816,\n        940.1176,\n        917.9796,\n        894.5576,\n        871.1858,\n        849.9144,\n        827.1142,\n        805.0818,\n        783.9664,\n        763.9096,\n        742.0816,\n        724.3962,\n        706.3454,\n        688.018,\n        667.4214,\n        650.3106,\n        633.0686,\n        613.8094,\n        597.818,\n        581.4248,\n        563.834,\n        547.363,\n        531.5066,\n        520.455400000001,\n        505.583199999999,\n        488.366,\n        476.480799999999,\n        459.7682,\n        450.0522,\n        434.328799999999,\n        423.952799999999,\n        408.727000000001,\n        399.079400000001,\n        387.252200000001,\n        373.987999999999,\n        360.852000000001,\n        351.6394,\n        339.642,\n        330.902400000001,\n        322.661599999999,\n        311.662200000001,\n        301.3254,\n        291.7484,\n        279.939200000001,\n        276.7508,\n        263.215200000001,\n        254.811400000001,\n        245.5494,\n        242.306399999999,\n        234.8734,\n        223.787200000001,\n        217.7156,\n        212.0196,\n        200.793,\n        195.9748,\n        189.0702,\n        182.449199999999,\n        177.2772,\n        170.2336,\n        164.741,\n        158.613600000001,\n        155.311,\n        147.5964,\n        142.837,\n        137.3724,\n        132.0162,\n        130.0424,\n        121.9804,\n        120.451800000001,\n        114.8968,\n        111.585999999999,\n        105.933199999999,\n        101.705,\n        98.5141999999996,\n        95.0488000000005,\n        89.7880000000005,\n        91.4750000000004,\n        83.7764000000006,\n        80.9698000000008,\n        72.8574000000008,\n        73.1615999999995,\n        67.5838000000003,\n        62.6263999999992,\n        63.2638000000006,\n        66.0977999999996,\n        52.0843999999997,\n        58.9956000000002,\n        47.0912000000008,\n        46.4956000000002,\n        48.4383999999991,\n        47.1082000000006,\n        43.2392,\n        37.2759999999998,\n        40.0283999999992,\n        35.1864000000005,\n        35.8595999999998,\n        32.0998,\n        28.027,\n        23.6694000000007,\n        33.8266000000003,\n        26.3736000000008,\n        27.2008000000005,\n        21.3245999999999,\n        26.4115999999995,\n        23.4521999999997,\n        19.5013999999992,\n        19.8513999999996,\n        10.7492000000002,\n        18.6424000000006,\n        13.1265999999996,\n        18.2436000000016,\n        6.71860000000015,\n        3.39459999999963,\n        6.33759999999893,\n        7.76719999999841,\n        0.813999999998487,\n        3.82819999999992,\n        0.826199999999517,\n        8.07440000000133,\n        -1.59080000000176,\n        5.01780000000144,\n        0.455399999998917,\n        -0.24199999999837,\n        0.174800000000687,\n        -9.07640000000174,\n        -4.20160000000033,\n        -3.77520000000004,\n        -4.75179999999818,\n        -5.3724000000002,\n        -8.90680000000066,\n        -6.10239999999976,\n        -5.74120000000039,\n        -9.95339999999851,\n        -3.86339999999836,\n        -13.7304000000004,\n        -16.2710000000006,\n        -7.51359999999841,\n        -3.30679999999847,\n        -13.1339999999982,\n        -10.0551999999989,\n        -6.72019999999975,\n        -8.59660000000076,\n        -10.9307999999983,\n        -1.8775999999998,\n        -4.82259999999951,\n        -13.7788,\n        -21.6470000000008,\n        -10.6735999999983,\n        -15.7799999999988,\n    ],\n    // precision 13\n    &[\n        5907.5052,\n        5802.2672,\n        5697.347,\n        5593.5794,\n        5491.2622,\n        5390.5514,\n        5290.3376,\n        5191.6952,\n        5093.5988,\n        4997.3552,\n        4902.5972,\n        4808.3082,\n        4715.5646,\n        4624.109,\n        4533.8216,\n        4444.4344,\n        4356.3802,\n        4269.2962,\n        4183.3784,\n        4098.292,\n        4014.79,\n        3932.4574,\n        3850.6036,\n        3771.2712,\n        3691.7708,\n        3615.099,\n        3538.1858,\n        3463.4746,\n        3388.8496,\n        3315.6794,\n        3244.5448,\n        3173.7516,\n        3103.3106,\n        3033.6094,\n        2966.5642,\n        2900.794,\n        2833.7256,\n        2769.81,\n        2707.3196,\n        2644.0778,\n        2583.9916,\n        2523.4662,\n        2464.124,\n        2406.073,\n        2347.0362,\n        2292.1006,\n        2238.1716,\n        2182.7514,\n        2128.4884,\n        2077.1314,\n        2025.037,\n        1975.3756,\n        1928.933,\n        1879.311,\n        1831.0006,\n        1783.2144,\n        1738.3096,\n        1694.5144,\n        1649.024,\n        1606.847,\n        1564.7528,\n        1525.3168,\n        1482.5372,\n        1443.9668,\n        1406.5074,\n        1365.867,\n        1329.2186,\n        1295.4186,\n        1257.9716,\n        1225.339,\n        1193.2972,\n        1156.3578,\n        1125.8686,\n        1091.187,\n        1061.4094,\n        1029.4188,\n        1000.9126,\n        972.3272,\n        944.004199999999,\n        915.7592,\n        889.965,\n        862.834200000001,\n        840.4254,\n        812.598399999999,\n        785.924200000001,\n        763.050999999999,\n        741.793799999999,\n        721.466,\n        699.040799999999,\n        677.997200000002,\n        649.866999999998,\n        634.911800000002,\n        609.8694,\n        591.981599999999,\n        570.2922,\n        557.129199999999,\n        538.3858,\n        521.872599999999,\n        502.951400000002,\n        495.776399999999,\n        475.171399999999,\n        459.751,\n        439.995200000001,\n        426.708999999999,\n        413.7016,\n        402.3868,\n        387.262599999998,\n        372.0524,\n        357.050999999999,\n        342.5098,\n        334.849200000001,\n        322.529399999999,\n        311.613799999999,\n        295.848000000002,\n        289.273000000001,\n        274.093000000001,\n        263.329600000001,\n        251.389599999999,\n        245.7392,\n        231.9614,\n        229.7952,\n        217.155200000001,\n        208.9588,\n        199.016599999999,\n        190.839199999999,\n        180.6976,\n        176.272799999999,\n        166.976999999999,\n        162.5252,\n        151.196400000001,\n        149.386999999999,\n        133.981199999998,\n        130.0586,\n        130.164000000001,\n        122.053400000001,\n        110.7428,\n        108.1276,\n        106.232400000001,\n        100.381600000001,\n        98.7668000000012,\n        86.6440000000002,\n        79.9768000000004,\n        82.4722000000002,\n        68.7026000000005,\n        70.1186000000016,\n        71.9948000000004,\n        58.998599999999,\n        59.0492000000013,\n        56.9818000000014,\n        47.5338000000011,\n        42.9928,\n        51.1591999999982,\n        37.2740000000013,\n        42.7220000000016,\n        31.3734000000004,\n        26.8090000000011,\n        25.8934000000008,\n        26.5286000000015,\n        29.5442000000003,\n        19.3503999999994,\n        26.0760000000009,\n        17.9527999999991,\n        14.8419999999969,\n        10.4683999999979,\n        8.65899999999965,\n        9.86720000000059,\n        4.34139999999752,\n        -0.907800000000861,\n        -3.32080000000133,\n        -0.936199999996461,\n        -11.9916000000012,\n        -8.87000000000262,\n        -6.33099999999831,\n        -11.3366000000024,\n        -15.9207999999999,\n        -9.34659999999712,\n        -15.5034000000014,\n        -19.2097999999969,\n        -15.357799999998,\n        -28.2235999999975,\n        -30.6898000000001,\n        -19.3271999999997,\n        -25.6083999999973,\n        -24.409599999999,\n        -13.6385999999984,\n        -33.4473999999973,\n        -32.6949999999997,\n        -28.9063999999998,\n        -31.7483999999968,\n        -32.2935999999972,\n        -35.8329999999987,\n        -47.620600000002,\n        -39.0855999999985,\n        -33.1434000000008,\n        -46.1371999999974,\n        -37.5892000000022,\n        -46.8164000000033,\n        -47.3142000000007,\n        -60.2914000000019,\n        -37.7575999999972,\n    ],\n    // precision 14\n    &[\n        11816.475,\n        11605.0046,\n        11395.3792,\n        11188.7504,\n        10984.1814,\n        10782.0086,\n        10582.0072,\n        10384.503,\n        10189.178,\n        9996.2738,\n        9806.0344,\n        9617.9798,\n        9431.394,\n        9248.7784,\n        9067.6894,\n        8889.6824,\n        8712.9134,\n        8538.8624,\n        8368.4944,\n        8197.7956,\n        8031.8916,\n        7866.6316,\n        7703.733,\n        7544.5726,\n        7386.204,\n        7230.666,\n        7077.8516,\n        6926.7886,\n        6778.6902,\n        6631.9632,\n        6487.304,\n        6346.7486,\n        6206.4408,\n        6070.202,\n        5935.2576,\n        5799.924,\n        5671.0324,\n        5541.9788,\n        5414.6112,\n        5290.0274,\n        5166.723,\n        5047.6906,\n        4929.162,\n        4815.1406,\n        4699.127,\n        4588.5606,\n        4477.7394,\n        4369.4014,\n        4264.2728,\n        4155.9224,\n        4055.581,\n        3955.505,\n        3856.9618,\n        3761.3828,\n        3666.9702,\n        3575.7764,\n        3482.4132,\n        3395.0186,\n        3305.8852,\n        3221.415,\n        3138.6024,\n        3056.296,\n        2970.4494,\n        2896.1526,\n        2816.8008,\n        2740.2156,\n        2670.497,\n        2594.1458,\n        2527.111,\n        2460.8168,\n        2387.5114,\n        2322.9498,\n        2260.6752,\n        2194.2686,\n        2133.7792,\n        2074.767,\n        2015.204,\n        1959.4226,\n        1898.6502,\n        1850.006,\n        1792.849,\n        1741.4838,\n        1687.9778,\n        1638.1322,\n        1589.3266,\n        1543.1394,\n        1496.8266,\n        1447.8516,\n        1402.7354,\n        1361.9606,\n        1327.0692,\n        1285.4106,\n        1241.8112,\n        1201.6726,\n        1161.973,\n        1130.261,\n        1094.2036,\n        1048.2036,\n        1020.6436,\n        990.901400000002,\n        961.199800000002,\n        924.769800000002,\n        899.526400000002,\n        872.346400000002,\n        834.375,\n        810.432000000001,\n        780.659800000001,\n        756.013800000001,\n        733.479399999997,\n        707.923999999999,\n        673.858,\n        652.222399999999,\n        636.572399999997,\n        615.738599999997,\n        586.696400000001,\n        564.147199999999,\n        541.679600000003,\n        523.943599999999,\n        505.714599999999,\n        475.729599999999,\n        461.779600000002,\n        449.750800000002,\n        439.020799999998,\n        412.7886,\n        400.245600000002,\n        383.188199999997,\n        362.079599999997,\n        357.533799999997,\n        334.319000000003,\n        327.553399999997,\n        308.559399999998,\n        291.270199999999,\n        279.351999999999,\n        271.791400000002,\n        252.576999999997,\n        247.482400000001,\n        236.174800000001,\n        218.774599999997,\n        220.155200000001,\n        208.794399999999,\n        201.223599999998,\n        182.995600000002,\n        185.5268,\n        164.547400000003,\n        176.5962,\n        150.689599999998,\n        157.8004,\n        138.378799999999,\n        134.021200000003,\n        117.614399999999,\n        108.194000000003,\n        97.0696000000025,\n        89.6042000000016,\n        95.6030000000028,\n        84.7810000000027,\n        72.635000000002,\n        77.3482000000004,\n        59.4907999999996,\n        55.5875999999989,\n        50.7346000000034,\n        61.3916000000027,\n        50.9149999999936,\n        39.0384000000049,\n        58.9395999999979,\n        29.633600000001,\n        28.2032000000036,\n        26.0078000000067,\n        17.0387999999948,\n        9.22000000000116,\n        13.8387999999977,\n        8.07240000000456,\n        14.1549999999988,\n        15.3570000000036,\n        3.42660000000615,\n        6.24820000000182,\n        -2.96940000000177,\n        -8.79940000000352,\n        -5.97860000000219,\n        -14.4048000000039,\n        -3.4143999999942,\n        -13.0148000000045,\n        -11.6977999999945,\n        -25.7878000000055,\n        -22.3185999999987,\n        -24.409599999999,\n        -31.9756000000052,\n        -18.9722000000038,\n        -22.8678000000073,\n        -30.8972000000067,\n        -32.3715999999986,\n        -22.3907999999938,\n        -43.6720000000059,\n        -35.9038,\n        -39.7492000000057,\n        -54.1641999999993,\n        -45.2749999999942,\n        -42.2989999999991,\n        -44.1089999999967,\n        -64.3564000000042,\n        -49.9551999999967,\n        -42.6116000000038,\n    ],\n    // precision 15\n    &[\n        23634.0036,\n        23210.8034,\n        22792.4744,\n        22379.1524,\n        21969.7928,\n        21565.326,\n        21165.3532,\n        20770.2806,\n        20379.9892,\n        19994.7098,\n        19613.318,\n        19236.799,\n        18865.4382,\n        18498.8244,\n        18136.5138,\n        17778.8668,\n        17426.2344,\n        17079.32,\n        16734.778,\n        16397.2418,\n        16063.3324,\n        15734.0232,\n        15409.731,\n        15088.728,\n        14772.9896,\n        14464.1402,\n        14157.5588,\n        13855.5958,\n        13559.3296,\n        13264.9096,\n        12978.326,\n        12692.0826,\n        12413.8816,\n        12137.3192,\n        11870.2326,\n        11602.5554,\n        11340.3142,\n        11079.613,\n        10829.5908,\n        10583.5466,\n        10334.0344,\n        10095.5072,\n        9859.694,\n        9625.2822,\n        9395.7862,\n        9174.0586,\n        8957.3164,\n        8738.064,\n        8524.155,\n        8313.7396,\n        8116.9168,\n        7913.542,\n        7718.4778,\n        7521.65,\n        7335.5596,\n        7154.2906,\n        6968.7396,\n        6786.3996,\n        6613.236,\n        6437.406,\n        6270.6598,\n        6107.7958,\n        5945.7174,\n        5787.6784,\n        5635.5784,\n        5482.308,\n        5337.9784,\n        5190.0864,\n        5045.9158,\n        4919.1386,\n        4771.817,\n        4645.7742,\n        4518.4774,\n        4385.5454,\n        4262.6622,\n        4142.74679999999,\n        4015.5318,\n        3897.9276,\n        3790.7764,\n        3685.13800000001,\n        3573.6274,\n        3467.9706,\n        3368.61079999999,\n        3271.5202,\n        3170.3848,\n        3076.4656,\n        2982.38400000001,\n        2888.4664,\n        2806.4868,\n        2711.9564,\n        2634.1434,\n        2551.3204,\n        2469.7662,\n        2396.61139999999,\n        2318.9902,\n        2243.8658,\n        2171.9246,\n        2105.01360000001,\n        2028.8536,\n        1960.9952,\n        1901.4096,\n        1841.86079999999,\n        1777.54700000001,\n        1714.5802,\n        1654.65059999999,\n        1596.311,\n        1546.2016,\n        1492.3296,\n        1433.8974,\n        1383.84600000001,\n        1339.4152,\n        1293.5518,\n        1245.8686,\n        1193.50659999999,\n        1162.27959999999,\n        1107.19439999999,\n        1069.18060000001,\n        1035.09179999999,\n        999.679000000004,\n        957.679999999993,\n        925.300199999998,\n        888.099400000006,\n        848.638600000006,\n        818.156400000007,\n        796.748399999997,\n        752.139200000005,\n        725.271200000003,\n        692.216,\n        671.633600000001,\n        647.939799999993,\n        621.670599999998,\n        575.398799999995,\n        561.226599999995,\n        532.237999999998,\n        521.787599999996,\n        483.095799999996,\n        467.049599999998,\n        465.286399999997,\n        415.548599999995,\n        401.047399999996,\n        380.607999999993,\n        377.362599999993,\n        347.258799999996,\n        338.371599999999,\n        310.096999999994,\n        301.409199999995,\n        276.280799999993,\n        265.586800000005,\n        258.994399999996,\n        223.915999999997,\n        215.925399999993,\n        213.503800000006,\n        191.045400000003,\n        166.718200000003,\n        166.259000000005,\n        162.941200000001,\n        148.829400000002,\n        141.645999999993,\n        123.535399999993,\n        122.329800000007,\n        89.473399999988,\n        80.1962000000058,\n        77.5457999999926,\n        59.1056000000099,\n        83.3509999999951,\n        52.2906000000075,\n        36.3979999999865,\n        40.6558000000077,\n        42.0003999999899,\n        19.6630000000005,\n        19.7153999999864,\n        -8.38539999999921,\n        -0.692799999989802,\n        0.854800000000978,\n        3.23219999999856,\n        -3.89040000000386,\n        -5.25880000001052,\n        -24.9052000000083,\n        -22.6837999999989,\n        -26.4286000000138,\n        -34.997000000003,\n        -37.0216000000073,\n        -43.430400000012,\n        -58.2390000000014,\n        -68.8034000000043,\n        -56.9245999999985,\n        -57.8583999999973,\n        -77.3097999999882,\n        -73.2793999999994,\n        -81.0738000000129,\n        -87.4530000000086,\n        -65.0254000000132,\n        -57.296399999992,\n        -96.2746000000043,\n        -103.25,\n        -96.081600000005,\n        -91.5542000000132,\n        -102.465200000006,\n        -107.688599999994,\n        -101.458000000013,\n        -109.715800000005,\n    ],\n    // precision 16\n    &[\n        47270.,\n        46423.3584,\n        45585.7074,\n        44757.152,\n        43938.8416,\n        43130.9514,\n        42330.03,\n        41540.407,\n        40759.6348,\n        39988.206,\n        39226.5144,\n        38473.2096,\n        37729.795,\n        36997.268,\n        36272.6448,\n        35558.665,\n        34853.0248,\n        34157.4472,\n        33470.5204,\n        32793.5742,\n        32127.0194,\n        31469.4182,\n        30817.6136,\n        30178.6968,\n        29546.8908,\n        28922.8544,\n        28312.271,\n        27707.0924,\n        27114.0326,\n        26526.692,\n        25948.6336,\n        25383.7826,\n        24823.5998,\n        24272.2974,\n        23732.2572,\n        23201.4976,\n        22674.2796,\n        22163.6336,\n        21656.515,\n        21161.7362,\n        20669.9368,\n        20189.4424,\n        19717.3358,\n        19256.3744,\n        18795.9638,\n        18352.197,\n        17908.5738,\n        17474.391,\n        17052.918,\n        16637.2236,\n        16228.4602,\n        15823.3474,\n        15428.6974,\n        15043.0284,\n        14667.6278,\n        14297.4588,\n        13935.2882,\n        13578.5402,\n        13234.6032,\n        12882.1578,\n        12548.0728,\n        12219.231,\n        11898.0072,\n        11587.2626,\n        11279.9072,\n        10973.5048,\n        10678.5186,\n        10392.4876,\n        10105.2556,\n        9825.766,\n        9562.5444,\n        9294.2222,\n        9038.2352,\n        8784.848,\n        8533.2644,\n        8301.7776,\n        8058.30859999999,\n        7822.94579999999,\n        7599.11319999999,\n        7366.90779999999,\n        7161.217,\n        6957.53080000001,\n        6736.212,\n        6548.21220000001,\n        6343.06839999999,\n        6156.28719999999,\n        5975.15419999999,\n        5791.75719999999,\n        5621.32019999999,\n        5451.66,\n        5287.61040000001,\n        5118.09479999999,\n        4957.288,\n        4798.4246,\n        4662.17559999999,\n        4512.05900000001,\n        4364.68539999999,\n        4220.77720000001,\n        4082.67259999999,\n        3957.19519999999,\n        3842.15779999999,\n        3699.3328,\n        3583.01180000001,\n        3473.8964,\n        3338.66639999999,\n        3233.55559999999,\n        3117.799,\n        3008.111,\n        2909.69140000001,\n        2814.86499999999,\n        2719.46119999999,\n        2624.742,\n        2532.46979999999,\n        2444.7886,\n        2370.1868,\n        2272.45259999999,\n        2196.19260000001,\n        2117.90419999999,\n        2023.2972,\n        1969.76819999999,\n        1885.58979999999,\n        1833.2824,\n        1733.91200000001,\n        1682.54920000001,\n        1604.57980000001,\n        1556.11240000001,\n        1491.3064,\n        1421.71960000001,\n        1371.22899999999,\n        1322.1324,\n        1264.7892,\n        1196.23920000001,\n        1143.8474,\n        1088.67240000001,\n        1073.60380000001,\n        1023.11660000001,\n        959.036400000012,\n        927.433199999999,\n        906.792799999996,\n        853.433599999989,\n        841.873800000001,\n        791.1054,\n        756.899999999994,\n        704.343200000003,\n        672.495599999995,\n        622.790399999998,\n        611.254799999995,\n        567.283200000005,\n        519.406599999988,\n        519.188400000014,\n        495.312800000014,\n        451.350799999986,\n        443.973399999988,\n        431.882199999993,\n        392.027000000002,\n        380.924200000009,\n        345.128999999986,\n        298.901400000002,\n        287.771999999997,\n        272.625,\n        247.253000000026,\n        222.490600000019,\n        223.590000000026,\n        196.407599999977,\n        176.425999999978,\n        134.725199999986,\n        132.4804,\n        110.445599999977,\n        86.7939999999944,\n        56.7038000000175,\n        64.915399999998,\n        38.3726000000024,\n        37.1606000000029,\n        46.170999999973,\n        49.1716000000015,\n        15.3362000000197,\n        6.71639999997569,\n        -34.8185999999987,\n        -39.4476000000141,\n        12.6830000000191,\n        -12.3331999999937,\n        -50.6565999999875,\n        -59.9538000000175,\n        -65.1054000000004,\n        -70.7576000000117,\n        -106.325200000021,\n        -126.852200000023,\n        -110.227599999984,\n        -132.885999999999,\n        -113.897200000007,\n        -142.713800000027,\n        -151.145399999979,\n        -150.799200000009,\n        -177.756200000003,\n        -156.036399999983,\n        -182.735199999996,\n        -177.259399999981,\n        -198.663600000029,\n        -174.577600000019,\n        -193.84580000001,\n    ],\n    // precision 17\n    &[\n        94541.,\n        92848.811,\n        91174.019,\n        89517.558,\n        87879.9705,\n        86262.7565,\n        84663.5125,\n        83083.7435,\n        81521.7865,\n        79977.272,\n        78455.9465,\n        76950.219,\n        75465.432,\n        73994.152,\n        72546.71,\n        71115.2345,\n        69705.6765,\n        68314.937,\n        66944.2705,\n        65591.255,\n        64252.9485,\n        62938.016,\n        61636.8225,\n        60355.592,\n        59092.789,\n        57850.568,\n        56624.518,\n        55417.343,\n        54231.1415,\n        53067.387,\n        51903.526,\n        50774.649,\n        49657.6415,\n        48561.05,\n        47475.7575,\n        46410.159,\n        45364.852,\n        44327.053,\n        43318.4005,\n        42325.6165,\n        41348.4595,\n        40383.6265,\n        39436.77,\n        38509.502,\n        37594.035,\n        36695.939,\n        35818.6895,\n        34955.691,\n        34115.8095,\n        33293.949,\n        32465.0775,\n        31657.6715,\n        30877.2585,\n        30093.78,\n        29351.3695,\n        28594.1365,\n        27872.115,\n        27168.7465,\n        26477.076,\n        25774.541,\n        25106.5375,\n        24452.5135,\n        23815.5125,\n        23174.0655,\n        22555.2685,\n        21960.2065,\n        21376.3555,\n        20785.1925,\n        20211.517,\n        19657.0725,\n        19141.6865,\n        18579.737,\n        18081.3955,\n        17578.995,\n        17073.44,\n        16608.335,\n        16119.911,\n        15651.266,\n        15194.583,\n        14749.0495,\n        14343.4835,\n        13925.639,\n        13504.509,\n        13099.3885,\n        12691.2855,\n        12328.018,\n        11969.0345,\n        11596.5145,\n        11245.6355,\n        10917.6575,\n        10580.9785,\n        10277.8605,\n        9926.58100000001,\n        9605.538,\n        9300.42950000003,\n        8989.97850000003,\n        8728.73249999998,\n        8448.3235,\n        8175.31050000002,\n        7898.98700000002,\n        7629.79100000003,\n        7413.76199999999,\n        7149.92300000001,\n        6921.12650000001,\n        6677.1545,\n        6443.28000000003,\n        6278.23450000002,\n        6014.20049999998,\n        5791.20299999998,\n        5605.78450000001,\n        5438.48800000001,\n        5234.2255,\n        5059.6825,\n        4887.43349999998,\n        4682.935,\n        4496.31099999999,\n        4322.52250000002,\n        4191.42499999999,\n        4021.24200000003,\n        3900.64799999999,\n        3762.84250000003,\n        3609.98050000001,\n        3502.29599999997,\n        3363.84250000003,\n        3206.54849999998,\n        3079.70000000001,\n        2971.42300000001,\n        2867.80349999998,\n        2727.08100000001,\n        2630.74900000001,\n        2496.6165,\n        2440.902,\n        2356.19150000002,\n        2235.58199999999,\n        2120.54149999999,\n        2012.25449999998,\n        1933.35600000003,\n        1820.93099999998,\n        1761.54800000001,\n        1663.09350000002,\n        1578.84600000002,\n        1509.48149999999,\n        1427.3345,\n        1379.56150000001,\n        1306.68099999998,\n        1212.63449999999,\n        1084.17300000001,\n        1124.16450000001,\n        1060.69949999999,\n        1007.48849999998,\n        941.194499999983,\n        879.880500000028,\n        836.007500000007,\n        782.802000000025,\n        748.385499999975,\n        647.991500000004,\n        626.730500000005,\n        570.776000000013,\n        484.000500000024,\n        513.98550000001,\n        418.985499999952,\n        386.996999999974,\n        370.026500000036,\n        355.496999999974,\n        356.731499999994,\n        255.92200000002,\n        259.094000000041,\n        205.434499999974,\n        165.374500000034,\n        197.347500000033,\n        95.718499999959,\n        67.6165000000037,\n        54.6970000000438,\n        31.7395000000251,\n        -15.8784999999916,\n        8.42500000004657,\n        -26.3754999999655,\n        -118.425500000012,\n        -66.6629999999423,\n        -42.9745000000112,\n        -107.364999999991,\n        -189.839000000036,\n        -162.611499999999,\n        -164.964999999967,\n        -189.079999999958,\n        -223.931499999948,\n        -235.329999999958,\n        -269.639500000048,\n        -249.087999999989,\n        -206.475499999942,\n        -283.04449999996,\n        -290.667000000016,\n        -304.561499999953,\n        -336.784499999951,\n        -380.386500000022,\n        -283.280499999993,\n        -364.533000000054,\n        -389.059499999974,\n        -364.454000000027,\n        -415.748000000021,\n        -417.155000000028,\n    ],\n    // precision 18\n    &[\n        189083.,\n        185696.913,\n        182348.774,\n        179035.946,\n        175762.762,\n        172526.444,\n        169329.754,\n        166166.099,\n        163043.269,\n        159958.91,\n        156907.912,\n        153906.845,\n        150924.199,\n        147996.568,\n        145093.457,\n        142239.233,\n        139421.475,\n        136632.27,\n        133889.588,\n        131174.2,\n        128511.619,\n        125868.621,\n        123265.385,\n        120721.061,\n        118181.769,\n        115709.456,\n        113252.446,\n        110840.198,\n        108465.099,\n        106126.164,\n        103823.469,\n        101556.618,\n        99308.004,\n        97124.508,\n        94937.803,\n        92833.731,\n        90745.061,\n        88677.627,\n        86617.47,\n        84650.442,\n        82697.833,\n        80769.132,\n        78879.629,\n        77014.432,\n        75215.626,\n        73384.587,\n        71652.482,\n        69895.93,\n        68209.301,\n        66553.669,\n        64921.981,\n        63310.323,\n        61742.115,\n        60205.018,\n        58698.658,\n        57190.657,\n        55760.865,\n        54331.169,\n        52908.167,\n        51550.273,\n        50225.254,\n        48922.421,\n        47614.533,\n        46362.049,\n        45098.569,\n        43926.083,\n        42736.03,\n        41593.473,\n        40425.26,\n        39316.237,\n        38243.651,\n        37170.617,\n        36114.609,\n        35084.19,\n        34117.233,\n        33206.509,\n        32231.505,\n        31318.728,\n        30403.404,\n        29540.0550000001,\n        28679.236,\n        27825.862,\n        26965.216,\n        26179.148,\n        25462.08,\n        24645.952,\n        23922.523,\n        23198.144,\n        22529.128,\n        21762.4179999999,\n        21134.779,\n        20459.117,\n        19840.818,\n        19187.04,\n        18636.3689999999,\n        17982.831,\n        17439.7389999999,\n        16874.547,\n        16358.2169999999,\n        15835.684,\n        15352.914,\n        14823.681,\n        14329.313,\n        13816.897,\n        13342.874,\n        12880.882,\n        12491.648,\n        12021.254,\n        11625.392,\n        11293.7610000001,\n        10813.697,\n        10456.209,\n        10099.074,\n        9755.39000000001,\n        9393.18500000006,\n        9047.57900000003,\n        8657.98499999999,\n        8395.85900000005,\n        8033.,\n        7736.95900000003,\n        7430.59699999995,\n        7258.47699999996,\n        6924.58200000005,\n        6691.29399999999,\n        6357.92500000005,\n        6202.05700000003,\n        5921.19700000004,\n        5628.28399999999,\n        5404.96799999999,\n        5226.71100000001,\n        4990.75600000005,\n        4799.77399999998,\n        4622.93099999998,\n        4472.478,\n        4171.78700000001,\n        3957.46299999999,\n        3868.95200000005,\n        3691.14300000004,\n        3474.63100000005,\n        3341.67200000002,\n        3109.14000000001,\n        3071.97400000005,\n        2796.40399999998,\n        2756.17799999996,\n        2611.46999999997,\n        2471.93000000005,\n        2382.26399999997,\n        2209.22400000005,\n        2142.28399999999,\n        2013.96100000001,\n        1911.18999999994,\n        1818.27099999995,\n        1668.47900000005,\n        1519.65800000005,\n        1469.67599999998,\n        1367.13800000004,\n        1248.52899999998,\n        1181.23600000003,\n        1022.71900000004,\n        1088.20700000005,\n        959.03600000008,\n        876.095999999903,\n        791.183999999892,\n        703.337000000058,\n        731.949999999953,\n        586.86400000006,\n        526.024999999907,\n        323.004999999888,\n        320.448000000091,\n        340.672999999952,\n        309.638999999966,\n        216.601999999955,\n        102.922999999952,\n        19.2399999999907,\n        -0.114000000059605,\n        -32.6240000000689,\n        -89.3179999999702,\n        -153.497999999905,\n        -64.2970000000205,\n        -143.695999999996,\n        -259.497999999905,\n        -253.017999999924,\n        -213.948000000091,\n        -397.590000000084,\n        -434.006000000052,\n        -403.475000000093,\n        -297.958000000101,\n        -404.317000000039,\n        -528.898999999976,\n        -506.621000000043,\n        -513.205000000075,\n        -479.351000000024,\n        -596.139999999898,\n        -527.016999999993,\n        -664.681000000099,\n        -680.306000000099,\n        -704.050000000047,\n        -850.486000000034,\n        -757.43200000003,\n        -713.308999999892,\n    ],\n];\n"
  },
  {
    "path": "crates/hyperloglogplusplus/src/lib.rs",
    "content": "#[cfg(test)]\nextern crate quickcheck;\n#[cfg(test)]\n#[macro_use(quickcheck)]\nextern crate quickcheck_macros;\n\nuse std::{\n    hash::{BuildHasher, Hash},\n    marker::PhantomData,\n};\n\npub mod dense;\nmod hyperloglog_data;\npub mod registers;\npub mod sparse;\n\n#[derive(serde::Serialize, serde::Deserialize, Clone, PartialEq, Eq)]\npub struct HyperLogLog<'s, T: ?Sized, B> {\n    storage: HyperLogLogStorage<'s>,\n    pub buildhasher: B,\n    _pd: PhantomData<T>,\n}\n\n#[derive(serde::Serialize, serde::Deserialize, Clone, PartialEq, Eq)]\npub enum HyperLogLogStorage<'s> {\n    Sparse(sparse::Storage<'s>),\n    Dense(dense::Storage<'s>),\n}\n\nimpl<'s, T, B> HyperLogLog<'s, T, B> {\n    pub fn new(precision: u8, buildhasher: B) -> Self {\n        Self {\n            storage: HyperLogLogStorage::Sparse(sparse::Storage::new(precision)),\n            buildhasher,\n            _pd: PhantomData,\n        }\n    }\n\n    pub fn from_sparse_parts(\n        bytes: &'s [u8],\n        num_compressed: u64,\n        precision: u8,\n        buildhasher: B,\n    ) -> Self {\n        Self {\n            storage: HyperLogLogStorage::Sparse(sparse::Storage::from_parts(\n                bytes,\n                num_compressed,\n                precision,\n            )),\n            buildhasher,\n            _pd: PhantomData,\n        }\n    }\n\n    pub fn from_dense_parts(bytes: &'s [u8], precision: u8, buildhasher: B) -> Self {\n        Self {\n            storage: HyperLogLogStorage::Dense(dense::Storage::from_parts(bytes, precision)),\n            buildhasher,\n            _pd: PhantomData,\n        }\n    }\n\n    pub fn estimate_count(&mut self) -> u64 {\n        use HyperLogLogStorage::*;\n\n        match &mut self.storage {\n            Sparse(s) => s.estimate_count(),\n            Dense(s) => s.estimate_count(),\n        }\n    }\n\n    pub fn immutable_estimate_count(&self) -> u64 {\n        use HyperLogLogStorage::*;\n\n        match &self.storage {\n            Sparse(s) => s.immutable_estimate_count(),\n            Dense(s) => s.estimate_count(),\n        }\n    }\n\n    pub fn is_sparse(&self) -> bool {\n        use HyperLogLogStorage::*;\n\n        matches!(&self.storage, Sparse(..))\n    }\n\n    pub fn num_bytes(&self) -> usize {\n        use HyperLogLogStorage::*;\n\n        match &self.storage {\n            Sparse(s) => s.num_bytes(),\n            Dense(s) => s.num_bytes(),\n        }\n    }\n\n    pub fn to_parts(&mut self) -> &HyperLogLogStorage<'s> {\n        self.merge_all();\n        &self.storage\n    }\n\n    pub fn merge_all(&mut self) {\n        match &mut self.storage {\n            HyperLogLogStorage::Sparse(s) => s.merge_buffers(),\n            HyperLogLogStorage::Dense(_) => {}\n        }\n    }\n\n    pub fn into_owned(&self) -> HyperLogLog<'static, T, B>\n    where\n        B: Clone,\n    {\n        use HyperLogLogStorage::*;\n        let storage = match &self.storage {\n            Sparse(s) => Sparse(s.into_owned()),\n            Dense(s) => Dense(s.into_owned()),\n        };\n        HyperLogLog {\n            storage,\n            buildhasher: self.buildhasher.clone(),\n            _pd: PhantomData,\n        }\n    }\n}\n\nimpl<T, B> HyperLogLog<'_, T, B>\nwhere\n    T: Hash + ?Sized,\n    B: BuildHasher,\n{\n    pub fn add(&mut self, value: &T) {\n        use HyperLogLogStorage::*;\n\n        let hash = self.buildhasher.hash_one(value);\n        match &mut self.storage {\n            Sparse(s) => {\n                let overflowing = s.add_hash(hash);\n                if overflowing {\n                    let dense = s.to_dense();\n                    self.storage = Dense(dense);\n                }\n            }\n            Dense(s) => s.add_hash(hash),\n        }\n    }\n\n    pub fn merge_in(&mut self, other: &HyperLogLog<'_, T, B>) {\n        use HyperLogLogStorage::*;\n        match (&mut self.storage, &other.storage) {\n            (Sparse(s), Sparse(o)) => {\n                let overflowing = s.merge_in(o);\n                if overflowing {\n                    let dense = s.to_dense();\n                    self.storage = Dense(dense);\n                }\n            }\n            (Sparse(s), Dense(o)) => {\n                let mut dense = s.to_dense();\n                dense.merge_in(o);\n                self.storage = Dense(dense);\n            }\n            (Dense(s), Sparse(o)) => s.merge_in(&o.immutable_to_dense()),\n            (Dense(s), Dense(o)) => s.merge_in(o),\n        }\n    }\n}\n\npub(crate) trait Extractable:\n    Sized + Copy + std::ops::Shl<u8, Output = Self> + std::ops::Shr<u8, Output = Self>\n{\n    const NUM_BITS: u8;\n    fn extract_bits(&self, high: u8, low: u8) -> Self {\n        self.extract(high, high - low + 1)\n    }\n    fn extract(&self, high: u8, len: u8) -> Self {\n        (*self << (Self::NUM_BITS - 1 - high)) >> (Self::NUM_BITS - len)\n    }\n    fn q(&self) -> u8;\n}\n\nimpl Extractable for u64 {\n    const NUM_BITS: u8 = 64;\n    fn q(&self) -> u8 {\n        self.leading_zeros() as u8 + 1\n    }\n}\n\nimpl Extractable for u32 {\n    const NUM_BITS: u8 = 32;\n    fn q(&self) -> u8 {\n        self.leading_zeros() as u8 + 1\n    }\n}\n\npub fn error_for_precision(precision: u8) -> f64 {\n    1.04 / 2.0f64.powi(precision.into()).sqrt()\n}\n\npub fn precision_for_error(max_error: f64) -> u8 {\n    // error = 1.04/sqrt(number_of_registers)\n    // error*sqrt(number_of_registers) = 1.04\n    // sqrt(number_of_registers) = 1.04/error\n    // number_of_registers = (1.04/error)^2\n    let num_registers = (1.04f64 / max_error).powi(2);\n    let precision = num_registers.log2().ceil();\n    if !(4.0..=18.0).contains(&precision) {\n        panic!(\"derived precision is not valid, error should be in the range [0.26, 0.00203125]\")\n    }\n    precision as u8\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashSet;\n\n    use fnv::FnvBuildHasher;\n    use quickcheck::TestResult;\n\n    use super::*;\n\n    #[test]\n    fn test_asc_4_10k() {\n        let mut hll = HyperLogLog::new(4, FnvBuildHasher::default());\n        for i in 0..10_000 {\n            hll.add(&i);\n        }\n        assert_eq!(hll.estimate_count(), 11113);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 13);\n        assert!(hll.num_bytes() <= (1 << 4) * 6 / 8 + 1);\n    }\n\n    #[test]\n    fn test_asc_4_100k() {\n        let mut hll = HyperLogLog::new(4, FnvBuildHasher::default());\n        for i in 0..100_000 {\n            hll.add(&i);\n        }\n        assert_eq!(hll.estimate_count(), 108_048);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 13);\n        assert!(hll.num_bytes() <= (1 << 4) * 6 / 8 + 1);\n    }\n\n    #[test]\n    fn test_asc_4_500k() {\n        let mut hll = HyperLogLog::new(4, FnvBuildHasher::default());\n        for i in 0..500_000 {\n            hll.add(&i);\n        }\n\n        assert_eq!(hll.estimate_count(), 425_701);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 13);\n        assert!(hll.num_bytes() <= (1 << 4) * 6 / 8 + 1);\n    }\n\n    #[test]\n    fn test_asc_8_10k() {\n        let mut hll = HyperLogLog::new(8, FnvBuildHasher::default());\n        for i in 0..10_000 {\n            hll.add(&i);\n        }\n        assert_eq!(hll.estimate_count(), 10_536);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 193);\n        assert!(hll.num_bytes() <= (1 << 8) * 6 / 8 + 1);\n    }\n\n    #[test]\n    fn test_asc_8_100k() {\n        let mut hll = HyperLogLog::new(8, FnvBuildHasher::default());\n        for i in 0..100_000 {\n            hll.add(&i);\n        }\n        assert_eq!(hll.estimate_count(), 121_578);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 193);\n        assert!(hll.num_bytes() <= (1 << 8) * 6 / 8 + 1);\n    }\n\n    #[test]\n    fn test_asc_8_500k() {\n        let mut hll = HyperLogLog::new(8, FnvBuildHasher::default());\n        for i in 0..500_000 {\n            hll.add(&i);\n        }\n\n        assert_eq!(hll.estimate_count(), 517_382);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 193);\n        assert!(hll.num_bytes() <= (1 << 8) * 6 / 8 + 1);\n    }\n\n    #[test]\n    fn test_asc_16_10k() {\n        let mut hll = HyperLogLog::new(16, FnvBuildHasher::default());\n        for i in 0..10_000 {\n            hll.add(&i);\n        }\n        assert_eq!(hll.estimate_count(), 10_001);\n        assert!(hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 23_181);\n        assert!(hll.num_bytes() <= (1 << 16) * 6 / 8 + 1)\n    }\n\n    #[test]\n    fn test_asc_16_100k() {\n        let mut hll = HyperLogLog::new(16, FnvBuildHasher::default());\n        for i in 0..100_000 {\n            hll.add(&i);\n        }\n        assert_eq!(hll.estimate_count(), 117_304);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 49_153);\n        assert!(hll.num_bytes() <= (1 << 16) * 6 / 8 + 1)\n    }\n\n    #[test]\n    fn test_asc_16_500k() {\n        let mut hll = HyperLogLog::new(16, FnvBuildHasher::default());\n        for i in 0..500_000 {\n            hll.add(&i);\n        }\n\n        assert_eq!(hll.estimate_count(), 510_445);\n        assert!(!hll.is_sparse());\n        assert_eq!(hll.num_bytes(), 49_153);\n        assert!(hll.num_bytes() <= (1 << 16) * 6 / 8 + 1)\n    }\n\n    #[quickcheck]\n    fn quick_hll_16(values: HashSet<u64>) -> TestResult {\n        let mut hll = HyperLogLog::new(16, FnvBuildHasher::default());\n        let expected = values.len() as f64;\n        for value in values {\n            hll.add(&value);\n        }\n        let estimated = hll.estimate_count() as f64;\n        let error = 0.0005 * expected;\n        if expected - error <= estimated && estimated <= expected + error {\n            return TestResult::passed();\n        }\n        if expected - 10.0 <= estimated && estimated <= expected + 10.0 {\n            return TestResult::passed();\n        }\n        println!(\"got {}, expected {} +- {}\", estimated, expected, error);\n        TestResult::failed()\n    }\n\n    #[quickcheck]\n    fn quick_merge_hll_16(values_a: Vec<u64>, values_b: Vec<u64>) {\n        let mut hll_a = HyperLogLog::new(16, FnvBuildHasher::default());\n        let mut baseline = HyperLogLog::new(16, FnvBuildHasher::default());\n        for value in values_a {\n            hll_a.add(&value);\n            baseline.add(&value)\n        }\n\n        let mut hll_b = HyperLogLog::new(16, FnvBuildHasher::default());\n        for value in values_b {\n            hll_b.add(&value);\n            baseline.add(&value)\n        }\n\n        hll_a.merge_all();\n        hll_b.merge_in(&hll_a);\n        assert_eq!(hll_b.estimate_count(), baseline.estimate_count())\n    }\n\n    // FIXME needs hash collision check\n    #[cfg(feature = \"flaky_tests\")]\n    #[quickcheck]\n    fn quick_merge_hll_8(values_a: Vec<u64>, values_b: Vec<u64>) {\n        let mut hll_a = HyperLogLog::new(8, FnvBuildHasher::default());\n        let mut baseline = HyperLogLog::new(8, FnvBuildHasher::default());\n        for value in &values_a {\n            hll_a.add(value);\n            baseline.add(value)\n        }\n\n        let mut hll_b = HyperLogLog::new(8, FnvBuildHasher::default());\n        for value in &values_b {\n            hll_b.add(value);\n            baseline.add(value)\n        }\n\n        hll_a.merge_all();\n        hll_b.merge_in(&hll_a);\n        let estimate = hll_b.estimate_count();\n        let baseline = baseline.estimate_count();\n        // FIXME\n        // if there's a hash collision between the elements unique to a and b\n        // the counts could be off slightly, check if there is in fact such a\n        // collision\n        if estimate > baseline + 5 || estimate < baseline.saturating_sub(6) {\n            panic!(\"{} != {}\", estimate, baseline)\n        }\n    }\n\n    #[quickcheck]\n    fn quick_merge_hll_4(values_a: Vec<u64>, values_b: Vec<u64>) {\n        let mut hll_a = HyperLogLog::new(4, FnvBuildHasher::default());\n        let mut baseline = HyperLogLog::new(4, FnvBuildHasher::default());\n        for value in values_a {\n            hll_a.add(&value);\n            baseline.add(&value)\n        }\n\n        let mut hll_b = HyperLogLog::new(4, FnvBuildHasher::default());\n        for value in values_b {\n            hll_b.add(&value);\n            baseline.add(&value)\n        }\n\n        hll_a.merge_all();\n        hll_b.merge_in(&hll_a);\n        assert_eq!(hll_b.estimate_count(), baseline.estimate_count())\n    }\n\n    #[test]\n    fn precision_for_error() {\n        for precision in 4..=18 {\n            assert_eq!(\n                super::precision_for_error(super::error_for_precision(precision)),\n                precision\n            )\n        }\n    }\n}\n"
  },
  {
    "path": "crates/hyperloglogplusplus/src/registers.rs",
    "content": "use std::{borrow::Cow, convert::TryInto, debug_assert};\n\n/// array of 6bit registers, of power-of-2 size\n// 24 is the LCM of 6 and 8, so we can divide our registers into\n// blocks of 24 bits and only deal with whole registers as follows:\n//\n//    b b b b b b|b b\n//    b b b b|b b b b\n//    b b|b b b b b b\n//\n// (3 bytes makes 4 whole registers)\n// We can turn this into a 32bit block like so\n//\n//    b b b b b b|b b\n//    b b b b|b b b b\n//    b b|b b b b b b\n//    0 0 0 0 0 0 0 0\n//\n// and treat the block like a regular integer, using shifts to get the\n// values in and out\n#[derive(Clone, serde::Serialize, serde::Deserialize, PartialEq, Eq)]\npub struct Registers<'s>(Cow<'s, [u8]>);\n\nimpl<'s> Registers<'s> {\n    /// allocate a new Registers of size `2^exponent`\n    pub fn new(exponent: u8) -> Self {\n        assert!((4..=64).contains(&exponent));\n        let num_registers: i128 = 1 << exponent;\n        let num_bits = num_registers * 6;\n        // store an additional byte at the end so we can always use 16-bit reads\n        // exhaustive search of the [4, 64] parameter space shows that this\n        // formula works correctly\n        let num_bytes = (num_bits / 8) + 1;\n        let mut bytes = vec![0u8; num_bytes as usize];\n\n        // set the extra byte to 0xff so we don't count it as 0\n        if let Some(byte) = bytes.last_mut() {\n            *byte = 0xff;\n        }\n\n        Self(bytes.into())\n    }\n\n    pub fn from_raw(bytes: &'s [u8]) -> Self {\n        Self(bytes.into())\n    }\n\n    #[cfg(test)]\n    pub fn at(&self, idx: usize) -> u8 {\n        // TODO switch chunks_exact_mut() to as_chunks_mut() once stable?\n        let block_num = idx / 4;\n        let idx_in_block = idx % 4;\n        let block = self.0.chunks_exact(3).nth(block_num).unwrap();\n        let block = u32::from_be_bytes([block[0], block[1], block[2], 0x0]);\n        let value = block >> (8 + 6 * (3 - idx_in_block));\n        (value & 0x3f) as u8\n    }\n\n    pub fn set_max(&mut self, idx: usize, value: u8) {\n        debug_assert!(value < (1 << 6));\n\n        let block_num = idx / 4;\n        let idx_in_block = idx % 4;\n        // TODO switch chunks_exact_mut() to as_chunks_mut() once stable?\n        let (a, b, c) = match self.0.to_mut().chunks_exact_mut(3).nth(block_num) {\n            Some([a, b, c, ..]) => (a, b, c),\n            _ => panic!(\n                \"index {} out of bounds of {} registers\",\n                idx,\n                (self.0.len() - 1) / 3 * 4,\n            ),\n        };\n\n        let block = u32::from_be_bytes([*a, *b, *c, 0x0]);\n\n        let shift = 8 // extra 0 byte at the end\n            + 6 * (3 - idx_in_block); // idx 0 is at the largest offset, so it needs the greatest shift\n        let mask = 0x3f << shift;\n        let value = (value as u32) << shift;\n\n        let old_value = block & mask;\n        if old_value < value {\n            let block = (block & !mask) | value;\n\n            let [new_a, new_b, new_c, _] = u32::to_be_bytes(block);\n\n            *a = new_a;\n            *b = new_b;\n            *c = new_c;\n        }\n    }\n\n    pub fn bytes(&self) -> &[u8] {\n        &self.0\n    }\n\n    pub fn count_zeroed_registers(&self) -> u64 {\n        self.iter().filter(|&b| b == 0).count() as u64\n    }\n\n    pub fn iter(&self) -> impl Iterator<Item = u8> + '_ {\n        use std::iter::once;\n\n        // our length should be divisible by 3, plus an extra byte we add\n        debug_assert_eq!(self.0.len() % 3, 1);\n\n        self.0.chunks_exact(3).flat_map(|bytes| {\n            const LOW_REG_MASK: u32 = (1 << 6) - 1;\n            let [a, b, c]: [u8; 3] = bytes.try_into().unwrap();\n            let block = u32::from_be_bytes([a, b, c, 0x0]);\n            // TODO replace with\n            // ```\n            // std::array::IntoIter::new([\n            //     ((block >> 26) & LOW_REG_MASK) as u8,\n            //     ((block >> 20) & LOW_REG_MASK) as u8,\n            //     ((block >> 14) & LOW_REG_MASK) as u8,\n            //     ((block >> 8) & LOW_REG_MASK) as u8,\n            // ])\n            // ```\n            // once std::array::IntoIter becomes stable\n            once(((block >> 26) & LOW_REG_MASK) as u8)\n                .chain(once(((block >> 20) & LOW_REG_MASK) as u8))\n                .chain(once(((block >> 14) & LOW_REG_MASK) as u8))\n                .chain(once(((block >> 8) & LOW_REG_MASK) as u8))\n        })\n    }\n\n    pub fn byte_len(&self) -> usize {\n        self.0.len()\n    }\n\n    pub fn merge(a: &Registers<'_>, b: &Registers<'_>) -> Self {\n        if a.0.len() != b.0.len() {\n            panic!(\n                \"different register size in merge: {} != {}\",\n                a.0.len(),\n                b.0.len()\n            )\n        }\n\n        let registers: Vec<u8> = (&*a.0).into();\n        let mut merged = Registers(registers.into());\n        for (i, v) in b.iter().enumerate() {\n            merged.set_max(i, v);\n        }\n\n        merged\n    }\n\n    pub fn into_owned(&self) -> Registers<'static> {\n        Registers(Cow::from(self.0.clone().into_owned()))\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn test_last_index_not_clobbered() {\n        for i in 4..14 {\n            let mut regs = Registers::new(i);\n\n            let read = regs.at((i - 1) as _);\n            assert!(read == 0, \"{}: {} = {}\", i, read, 0);\n\n            regs.set_max((i - 1) as _, 0xf);\n            let read = regs.at((i - 1) as _);\n            assert!(read == 0xf, \"{}: {} = {}\", i, read, 0xf);\n\n            if i > 1 {\n                let read = regs.at((i - 2) as _);\n                assert!(read == 0, \"{}: {} = {}\", i, read, 0);\n\n                regs.set_max((i - 2) as _, 0x3f);\n                let read = regs.at((i - 2) as _);\n                assert!(read == 0x3f, \"{}: {} = {}\", i, read, 0x3f);\n\n                let read = regs.at((i - 1) as _);\n                assert!(read == 0xf, \"{}: {} = {}\", i, read, 0xf);\n            }\n        }\n    }\n\n    #[test]\n    fn test_last_index_not_clobbers() {\n        for i in 4..14 {\n            let mut regs = Registers::new(i);\n\n            let read = regs.at((i - 2) as _);\n            assert!(read == 0, \"{}: {} = {}\", i, read, 0);\n\n            regs.set_max((i - 2) as _, 0x3c);\n            let read = regs.at((i - 2) as _);\n            assert!(read == 0x3c, \"{}: {} = {}\", i, read, 0x3c);\n\n            let read = regs.at((i - 1) as _);\n            assert!(read == 0, \"{}: {} = {}\", i, read, 0);\n\n            let read = regs.at((i - 1) as _);\n            assert!(read == 0, \"{}: {} = {}\", i, read, 0);\n\n            regs.set_max((i - 1) as _, 0x3f);\n            let read = regs.at((i - 1) as _);\n            assert!(read == 0x3f, \"{}: {} = {}\", i, read, 0x3f);\n\n            if i > 1 {\n                let read = regs.at((i - 2) as _);\n                assert!(read == 0x3c, \"{}: {} = {}\", i, read, 0x3c);\n            }\n        }\n    }\n\n    #[test]\n    fn test_count_empty() {\n        assert_eq!(Registers::new(4).count_zeroed_registers(), 16);\n    }\n\n    #[test]\n    fn test_count_4() {\n        let registers = Registers::new(4);\n        assert_eq!(registers.count_zeroed_registers(), 16);\n    }\n\n    #[test]\n    fn test_count_5() {\n        let registers = Registers::new(5);\n        assert_eq!(registers.count_zeroed_registers(), 32);\n    }\n\n    #[test]\n    fn test_count_6() {\n        let registers = Registers::new(6);\n        assert_eq!(registers.count_zeroed_registers(), 64);\n    }\n\n    #[test]\n    fn test_count_7() {\n        let registers = Registers::new(7);\n        assert_eq!(registers.count_zeroed_registers(), 128);\n    }\n\n    #[test]\n    fn test_iter_4_0_1() {\n        let mut registers = Registers::new(4);\n        registers.set_max(0, 1);\n        let values: Vec<_> = registers.iter().collect();\n        let mut expected = [0; 16];\n        expected[0] = 1;\n        assert_eq!(values, expected);\n    }\n\n    #[quickcheck]\n    fn quick_test(exp: u8, ops: Vec<(usize, u8)>) -> quickcheck::TestResult {\n        use quickcheck::TestResult;\n        use std::cmp::max;\n        if !(4..=16).contains(&exp) {\n            return TestResult::discard();\n        }\n\n        let size = 1 << exp;\n        let mut reference = vec![0; size];\n        let mut registers = Registers::new(exp);\n        for (idx, val) in ops {\n            let fixed_idx = idx % size;\n            let val = val & 0x3f;\n            reference[fixed_idx] = max(val, reference[fixed_idx]);\n            registers.set_max(fixed_idx, val);\n        }\n        let mut expected_count = 0;\n        for (idx, val) in reference.iter().enumerate() {\n            if registers.at(idx) != *val {\n                return TestResult::failed();\n            }\n            if *val == 0 {\n                expected_count += 1;\n            }\n        }\n\n        let expeceted_len = reference.len();\n        let mut actual_len = 0;\n        for (i, (a, b)) in reference.iter().zip(registers.iter()).enumerate() {\n            if *a != b {\n                println!(\"value mismatch @ {}, expected {}, got {}\", i, a, b,);\n                return TestResult::failed();\n            }\n            actual_len += 1\n        }\n        if expeceted_len != actual_len {\n            println!(\n                \"iter len mismatch, expected {}, got {}\",\n                expeceted_len, actual_len,\n            );\n            return TestResult::failed();\n        }\n\n        let actual_count = registers.count_zeroed_registers();\n        if actual_count != expected_count {\n            println!(\n                \"count mismatch, expected {}, got {}\",\n                expected_count, actual_count,\n            );\n            return TestResult::failed();\n        }\n        TestResult::passed()\n    }\n\n    #[quickcheck]\n    fn quick_merge(\n        exp: u8,\n        ops_a: Vec<(usize, u8)>,\n        ops_b: Vec<(usize, u8)>,\n    ) -> quickcheck::TestResult {\n        use quickcheck::TestResult;\n        if !(4..=16).contains(&exp) {\n            return TestResult::discard();\n        }\n\n        let size = 1 << exp;\n        let mut reference = Registers::new(exp);\n        let mut a = Registers::new(exp);\n        for (idx, val) in ops_a {\n            let fixed_idx = idx % size;\n            let val = val & 0x3f;\n            a.set_max(fixed_idx, val);\n            reference.set_max(fixed_idx, val);\n        }\n\n        let mut b = Registers::new(exp);\n        for (idx, val) in ops_b {\n            let fixed_idx = idx % size;\n            let val = val & 0x3f;\n            b.set_max(fixed_idx, val);\n            reference.set_max(fixed_idx, val);\n        }\n\n        let merged = Registers::merge(&a, &b);\n        assert_eq!(&*merged.0, &*reference.0);\n        TestResult::passed()\n    }\n}\n"
  },
  {
    "path": "crates/hyperloglogplusplus/src/sparse/varint.rs",
    "content": "use std::borrow::Cow;\n\nuse encodings::{delta, prefix_varint};\n\nuse super::Encoded;\n\npub fn decompression_iter<'a>(\n    Compressed(bytes): &'a Compressed<'_>,\n) -> impl Iterator<Item = Encoded> + 'a {\n    prefix_varint::u64_decompressor(bytes)\n        .map(delta::u64_decoder())\n        .map(|v| Encoded(v as u32))\n}\n\n#[derive(Default, serde::Serialize, serde::Deserialize, Clone, PartialEq, Eq)]\npub struct Compressed<'c>(Cow<'c, [u8]>);\n\nimpl<'c> Compressed<'c> {\n    pub fn from_raw(bytes: &'c [u8]) -> Self {\n        Self(bytes.into())\n    }\n\n    pub fn bytes(&self) -> &[u8] {\n        &self.0\n    }\n\n    pub fn num_bytes(&self) -> usize {\n        self.0.len()\n    }\n\n    #[allow(dead_code)]\n    pub fn cap(&self) -> usize {\n        self.0.len()\n    }\n\n    pub fn make_owned(&self) -> Compressed<'static> {\n        Compressed(Cow::from(self.0.clone().into_owned()))\n    }\n}\n\npub struct Compressor<F: FnMut(u64) -> u64> {\n    compressor: prefix_varint::U64Compressor<F>,\n    buffer: Option<Encoded>,\n    num_compressed: u64,\n}\n\n// TODO add capacity\npub fn compressor() -> Compressor<impl FnMut(u64) -> u64> {\n    Compressor {\n        compressor: prefix_varint::U64Compressor::with(delta::u64_encoder()),\n        buffer: None,\n        num_compressed: 0,\n    }\n}\n\nimpl<F: FnMut(u64) -> u64> Compressor<F> {\n    pub fn is_empty(&self) -> bool {\n        self.buffer.is_none() && self.compressor.is_empty()\n    }\n\n    pub fn last_mut(&mut self) -> Option<&mut Encoded> {\n        self.buffer.as_mut()\n    }\n\n    pub fn push(&mut self, value: Encoded) {\n        if let Some(val) = self.buffer.take() {\n            self.compress_value(val)\n        }\n\n        self.buffer = Some(value)\n    }\n\n    pub fn into_compressed(mut self) -> (Compressed<'static>, u64) {\n        if let Some(val) = self.buffer.take() {\n            self.compress_value(val)\n        }\n\n        (\n            Compressed(self.compressor.finish().into()),\n            self.num_compressed,\n        )\n    }\n\n    fn compress_value(&mut self, Encoded(value): Encoded) {\n        self.num_compressed += 1;\n        self.compressor.push(value.into());\n    }\n}\n\nimpl<F: FnMut(u64) -> u64> Extend<Encoded> for Compressor<F> {\n    fn extend<T: IntoIterator<Item = Encoded>>(&mut self, iter: T) {\n        for e in iter {\n            self.push(e)\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[quickcheck]\n    fn quick_test_roundtrip(values: Vec<u32>) -> bool {\n        let mut compressor = compressor();\n        for val in &values {\n            compressor.push(Encoded(*val));\n        }\n        let (blocks, count) = compressor.into_compressed();\n\n        let decompressed = decompression_iter(&blocks);\n\n        let expected_len = values.len();\n        let mut actual_len = 0;\n        for (i, (a, b)) in values.iter().zip(decompressed).enumerate() {\n            if *a != b.0 {\n                println!(\"value mismatch @ {}, expected {}, got {}\", i, a, b.0,);\n                return false;\n            }\n            actual_len += 1\n        }\n\n        if expected_len != actual_len {\n            println!(\n                \"iter len mismatch, expected {}, got {}\",\n                expected_len, actual_len,\n            );\n            return false;\n        }\n        if expected_len as u64 != count {\n            println!(\n                \"compression count mismatch, expected {}, got {}\",\n                expected_len, count,\n            );\n            return false;\n        }\n        true\n    }\n}\n"
  },
  {
    "path": "crates/hyperloglogplusplus/src/sparse.rs",
    "content": "use std::{\n    cmp::{\n        min,\n        Ordering::{Equal, Greater, Less},\n    },\n    collections::HashSet,\n};\n\nuse crate::{dense, Extractable};\n\nuse self::varint::*;\n\nmod varint;\n\n#[derive(serde::Serialize, serde::Deserialize, Clone, PartialEq, Eq)]\npub struct Storage<'s> {\n    to_merge: HashSet<Encoded>,\n    pub compressed: Compressed<'s>,\n    pub num_compressed: u64,\n    pub precision: u8,\n}\n\n#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq, serde::Serialize, serde::Deserialize)]\n#[repr(transparent)]\npub struct Encoded(u32);\n\nconst NUM_HIGH_BITS: u8 = 25;\n\npub type Overflowing = bool;\n\nimpl<'s> Storage<'s> {\n    pub fn new(precision: u8) -> Self {\n        // TODO what is max precision\n        assert!(\n            (4..=18).contains(&precision),\n            \"invalid value for precision: {precision}; must be within [4, 18]\",\n        );\n        Self {\n            to_merge: Default::default(),\n            compressed: Default::default(),\n            num_compressed: 0,\n            precision,\n        }\n    }\n\n    pub fn from_parts(bytes: &'s [u8], num_compressed: u64, precision: u8) -> Self {\n        // TODO what is max precision\n        assert!(\n            (4..=18).contains(&precision),\n            \"invalid value for precision: {precision}; must be within [4, 18]\",\n        );\n        Self {\n            to_merge: Default::default(),\n            compressed: Compressed::from_raw(bytes),\n            num_compressed,\n            precision,\n        }\n    }\n\n    pub fn into_owned(&self) -> Storage<'static> {\n        Storage {\n            to_merge: self.to_merge.clone(),\n            compressed: self.compressed.make_owned(),\n            num_compressed: self.num_compressed,\n            precision: self.precision,\n        }\n    }\n\n    pub fn add_hash(&mut self, hash: u64) -> Overflowing {\n        let encoded = Encoded::from_hash(hash, self.precision);\n        self.add_encoded(encoded)\n    }\n\n    fn add_encoded(&mut self, encoded: Encoded) -> Overflowing {\n        self.to_merge.insert(encoded);\n        let max_sparse_bitsize = (1u64 << self.precision) * 6;\n        // TODO what threshold?\n        if self.to_merge.len() as u64 * 32 > max_sparse_bitsize / 4 {\n            self.merge_buffers();\n            return self.compressed.num_bytes() as u64 * 8 > max_sparse_bitsize;\n        }\n        false\n    }\n\n    pub fn estimate_count(&mut self) -> u64 {\n        self.merge_buffers();\n        self.immutable_estimate_count()\n    }\n\n    pub fn immutable_estimate_count(&self) -> u64 {\n        if !self.to_merge.is_empty() {\n            panic!(\"tried to estimate count with unmerged state\")\n        }\n        let m_p = 1 << NUM_HIGH_BITS;\n        let v = (m_p - self.num_compressed) as f64;\n        let m_p = m_p as f64;\n        (m_p * (m_p / v).ln()) as u64\n    }\n\n    pub fn merge_buffers(&mut self) {\n        if self.to_merge.is_empty() {\n            return;\n        }\n        let mut temp: Vec<_> = self.to_merge.drain().collect();\n        temp.sort_unstable();\n        temp.dedup_by_key(|e| e.idx());\n\n        // TODO set original cap to self.compressed.cap()\n        let mut new_compressed = compressor();\n        let mut a = decompression_iter(&self.compressed).peekable();\n        let mut b = temp.into_iter().fuse().peekable();\n\n        let mut merge_in = |to_merge_in| {\n            if new_compressed.is_empty() {\n                new_compressed.push(to_merge_in);\n                return;\n            }\n\n            let prev = new_compressed.last_mut().unwrap();\n            if prev.idx() != to_merge_in.idx() {\n                new_compressed.push(to_merge_in);\n                return;\n            }\n\n            if prev.count(NUM_HIGH_BITS) < to_merge_in.count(NUM_HIGH_BITS) {\n                *prev = to_merge_in;\n            }\n        };\n        while let (Some(val_a), Some(val_b)) = (a.peek(), b.peek()) {\n            let (idx_a, idx_b) = (val_a.idx(), val_b.idx());\n            let to_merge_in = match idx_a.cmp(&idx_b) {\n                Less => a.next().unwrap(),\n                Greater => b.next().unwrap(),\n                Equal => {\n                    let (a, b) = (a.next().unwrap(), b.next().unwrap());\n                    min(a, b)\n                }\n            };\n            merge_in(to_merge_in);\n        }\n        a.for_each(&mut merge_in);\n        b.for_each(merge_in);\n\n        let (compressed, count) = new_compressed.into_compressed();\n        self.compressed = compressed;\n        self.num_compressed = count;\n    }\n\n    fn iter(&self) -> impl Iterator<Item = Encoded> + '_ {\n        decompression_iter(&self.compressed)\n    }\n\n    pub fn to_dense(&mut self) -> dense::Storage<'static> {\n        self.merge_buffers();\n\n        self.immutable_to_dense()\n    }\n\n    pub fn immutable_to_dense(&self) -> dense::Storage<'static> {\n        if !self.to_merge.is_empty() {\n            panic!(\"tried to generate dense storage with unmerged state\")\n        }\n        let mut dense = dense::Storage::new(self.precision);\n        for encoded in self.iter() {\n            dense.add_encoded(encoded)\n        }\n        dense\n    }\n\n    pub fn num_bytes(&self) -> usize {\n        self.compressed.num_bytes()\n    }\n\n    pub fn merge_in(&mut self, other: &Storage<'_>) -> Overflowing {\n        assert!(\n            self.precision == other.precision,\n            \"precision must be equal (left={}, right={})\",\n            self.precision,\n            other.precision\n        );\n\n        assert!(other.to_merge.is_empty());\n\n        let mut overflowing = false;\n        for encoded in other.iter() {\n            overflowing = self.add_encoded(encoded)\n        }\n        overflowing\n    }\n}\n\nimpl Encoded {\n    pub(crate) fn from_hash(hash: u64, precision: u8) -> Self {\n        // Encoded form\n        //\n        //    | idx | count | tag |\n        //    |  25 |   6*  |  1  |\n        //\n        // *`count` is only present when `tag` is `1`\n        let idx = hash.extract(63, NUM_HIGH_BITS) as u32;\n        let diff = hash.extract_bits(63 - precision, 64 - NUM_HIGH_BITS);\n        if diff == 0 {\n            // TODO is this right?\n            let count = hash.extract_bits(63 - NUM_HIGH_BITS, 0).q() as u32 - NUM_HIGH_BITS as u32;\n            Encoded((idx << 7) | (count << 1) | 1)\n        } else {\n            Encoded(idx << 1)\n        }\n    }\n\n    pub fn idx(&self) -> u32 {\n        if self.stores_count() {\n            self.0 >> 7\n        } else {\n            self.0 >> 1\n        }\n    }\n\n    pub fn count(&self, p: u8) -> u8 {\n        if self.stores_count() {\n            let extra_bits = NUM_HIGH_BITS - p;\n            self.extract_count() + extra_bits\n        } else {\n            let new_hash = (self.idx() as u64) << (64 - NUM_HIGH_BITS);\n            let hash_bits = new_hash.extract_bits(63 - p, 0);\n            hash_bits.q() - p\n        }\n    }\n\n    #[inline]\n    fn stores_count(&self) -> bool {\n        self.0 & 1 == 1\n    }\n\n    #[inline]\n    fn extract_count(&self) -> u8 {\n        self.0.extract_bits(6, 1) as u8\n    }\n}\n\nimpl PartialOrd for Encoded {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\n// The canonical ordering is by ascending index, then descending count.\n// This allows us to deduplicate by index after sorting.\nimpl Ord for Encoded {\n    fn cmp(&self, other: &Self) -> std::cmp::Ordering {\n        let idx_cmp = self.idx().cmp(&other.idx());\n        if let Equal = idx_cmp {\n            return match (self.stores_count(), other.stores_count()) {\n                (false, false) => Equal,\n                (true, false) => Less,\n                (false, true) => Greater,\n                (true, true) => self.extract_count().cmp(&other.extract_count()).reverse(),\n            };\n        }\n        idx_cmp\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use fnv::FnvHasher;\n    use quickcheck::TestResult;\n\n    use super::*;\n\n    use std::hash::{Hash, Hasher};\n\n    const NUM_HASH_BITS: u8 = 64 - NUM_HIGH_BITS;\n\n    pub fn hash(val: i32) -> u64 {\n        let mut hasher = FnvHasher::default();\n        val.hash(&mut hasher);\n        hasher.finish()\n    }\n\n    #[test]\n    fn test_asc_10k() {\n        let mut hll = Storage::new(16);\n        for i in 0..10_000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 10_001)\n    }\n\n    #[test]\n    fn test_asc_100k() {\n        let mut hll = Storage::new(16);\n        for i in 0..100_000 {\n            hll.add_hash(hash(i));\n        }\n        assert_eq!(hll.estimate_count(), 100_149);\n        assert_eq!(hll.compressed.num_bytes(), 184_315);\n    }\n\n    #[test]\n    fn test_asc_500k() {\n        let mut hll = Storage::new(16);\n        for i in 0..500_000 {\n            hll.add_hash(hash(i));\n        }\n\n        assert_eq!(hll.estimate_count(), 471_229);\n        assert_eq!(hll.compressed.num_bytes(), 690_301);\n    }\n\n    #[quickcheck]\n    fn quick_sparse(values: Vec<u64>) -> TestResult {\n        if values.len() >= (1 << NUM_HASH_BITS) {\n            return TestResult::discard();\n        }\n        let mut hll = Storage::new(16);\n        let expected = values.iter().collect::<HashSet<_>>().len() as f64;\n        for value in values {\n            hll.add_hash(value);\n        }\n        let estimated = hll.estimate_count() as f64;\n        let error = 0.001 * expected;\n        if expected - error <= estimated && estimated <= expected + error {\n            return TestResult::passed();\n        }\n        if estimated <= expected + 10.0 && estimated >= expected - 10.0 {\n            return TestResult::passed();\n        }\n        println!(\"got {}, expected {} +- {}\", estimated, expected, error);\n        TestResult::failed()\n    }\n\n    #[quickcheck]\n    fn quick_sparse_as_set(values: Vec<u64>) -> TestResult {\n        if values.len() >= (1 << NUM_HASH_BITS) {\n            return TestResult::discard();\n        }\n        let mut hll = Storage::new(16);\n        for value in &values {\n            hll.add_hash(*value);\n        }\n        hll.merge_buffers();\n        let mut expected: Vec<_> = values\n            .into_iter()\n            .map(|h| Encoded::from_hash(h, 16))\n            .collect();\n        expected.sort_unstable();\n        // println!(\"pre_sort {:?}\", temp);\n        expected.dedup_by_key(|e| e.idx());\n\n        let expected_len = expected.len();\n        let mut actual_len = 0;\n        for (i, (a, b)) in expected.iter().zip(hll.iter()).enumerate() {\n            if *a != b {\n                println!(\"value mismatch @ {}, expected {}, got {}\", i, a.0, b.0,);\n                return TestResult::failed();\n            }\n            actual_len += 1\n        }\n        if expected_len != actual_len {\n            println!(\n                \"iter len mismatch, expected {}, got {}\",\n                expected_len, actual_len,\n            );\n            return TestResult::failed();\n        }\n        TestResult::passed()\n    }\n\n    #[quickcheck]\n    fn quick_sparse_merge_invariant(values: Vec<u64>) -> TestResult {\n        if values.len() >= (1 << NUM_HASH_BITS) {\n            return TestResult::discard();\n        }\n        let mut hlla = Storage::new(16);\n        let mut hllb = Storage::new(16);\n        for value in &values {\n            hlla.add_hash(*value);\n            hllb.add_hash(*value);\n            hllb.merge_buffers()\n        }\n        hlla.merge_buffers();\n\n        for (i, (a, b)) in hlla.iter().zip(hllb.iter()).enumerate() {\n            if a != b {\n                println!(\"value mismatch @ {}, expected {}, got {}\", i, a.0, b.0,);\n                return TestResult::failed();\n            }\n        }\n\n        let expected_len = hlla.iter().count();\n        let actual_len = hllb.iter().count();\n        if expected_len != actual_len {\n            println!(\n                \"iter len mismatch, expected {}, got {}\",\n                expected_len, actual_len,\n            );\n            return TestResult::failed();\n        }\n        TestResult::passed()\n    }\n\n    // fn encoded_order() {\n\n    // }\n\n    #[test]\n    fn sparse_merge_01() {\n        let mut hlla = Storage::new(16);\n        let mut hllb = Storage::new(16);\n        let values = [0, 1];\n        for value in &values {\n            hlla.add_hash(*value);\n        }\n        hlla.merge_buffers();\n\n        for value in &values {\n            hllb.add_hash(*value);\n            hllb.merge_buffers()\n        }\n        let a: Vec<_> = hlla.iter().collect();\n        let b: Vec<_> = hllb.iter().collect();\n        assert_eq!(a, b)\n    }\n}\n"
  },
  {
    "path": "crates/scripting-utilities/Readme.md",
    "content": "# Scripting Utilities #\n\nSmall helper crates for writing scripty code, such as found in tools.\nContains code that's _just_ complicated or irritating enough that it's worth\ndeduplicating instead of copy/pasting, but still simple enough to be appropriate\nfor scripty code.\n\nWe care about compile times for this code, so in general try to keep the crates\nsmall, simple, and easy to understand. In accordance with this, this dir\ncontains a bunch of micro crates instead of one medium utility crate in hopes\nthat this will keep them more focused and prevent them from metastasizing, and\ntake advantage of compiler parallelism."
  },
  {
    "path": "crates/scripting-utilities/control_file_reader/Cargo.toml",
    "content": "[package]\nname = \"control_file_reader\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\n"
  },
  {
    "path": "crates/scripting-utilities/control_file_reader/src/lib.rs",
    "content": "/// Code to extract info from `timescaledb_toolkit.control`\n/// This crate exists so we have a single source of truth for the format.\nuse std::fmt;\n\npub type Result<T, E = Error> = std::result::Result<T, E>;\n\n/// extract the current version from the control file\npub fn get_current_version(control_file: &str) -> Result<String> {\n    get_field_val(control_file, \"version\").map(|v| v.to_string())\n}\n\n/// extract the list of versions we're upgradeable-from from the control file\npub fn get_upgradeable_from(control_file: &str) -> Result<Vec<String>> {\n    // versions is a comma-delimited list of versions\n    let versions = get_field_val(control_file, \"upgradeable_from\")?;\n    let versions = versions\n        .split_terminator(',')\n        .map(|version| version.trim().to_string())\n        .collect();\n    Ok(versions)\n}\n\n/// find a `<field name> = '<field value>'` in `file` and extract `<field value>`\npub fn get_field_val<'a>(file: &'a str, field_name: &str) -> Result<&'a str> {\n    file.lines()\n        .filter(|line| line.starts_with(field_name) || line.starts_with(&format!(\"# {field_name}\")))\n        .map(get_quoted_field)\n        .next()\n        .ok_or(Error::FieldNotFound)\n        .and_then(|e| e) // flatten the nested results\n}\n\n// given a `<field name> = '<field value>'` extract `<field value>`\npub fn get_quoted_field(line: &str) -> Result<&str> {\n    let quoted = line.split('=').nth(1).ok_or(Error::NoValue)?;\n\n    quoted\n        .trim_start()\n        .split_terminator('\\'')\n        .find(|s| !s.is_empty())\n        .ok_or(Error::UnquotedValue)\n}\n\npub enum Error {\n    FieldNotFound,\n    NoValue,\n    UnquotedValue,\n}\n\nimpl fmt::Display for Error {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        match &self {\n            Self::FieldNotFound => write!(f, \"cannot read field\"),\n            Self::NoValue => write!(f, \"cannot find value\"),\n            Self::UnquotedValue => write!(f, \"unquoted value\"),\n        }\n    }\n}\n\nimpl fmt::Debug for Error {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        fmt::Display::fmt(self, f)\n    }\n}\n"
  },
  {
    "path": "crates/scripting-utilities/postgres_connection_configuration/Cargo.toml",
    "content": "[package]\nname = \"postgres_connection_configuration\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\n"
  },
  {
    "path": "crates/scripting-utilities/postgres_connection_configuration/src/lib.rs",
    "content": "/// Config utility for connecting to multiple DBs in the same cluster\n// JOSH - I'm not sure if this really warrants a crate, but it seems like if we\n//        ever change this it'll be annoying to hunt down everything ¯\\_(ツ)_/¯\n\n#[derive(Copy, Clone)]\npub struct ConnectionConfig<'s> {\n    pub host: Option<&'s str>,\n    pub port: Option<&'s str>,\n    pub user: Option<&'s str>,\n    pub password: Option<&'s str>,\n    pub database: Option<&'s str>,\n}\n\nimpl<'s> ConnectionConfig<'s> {\n    pub fn with_db<'d>(&self, database: &'d str) -> ConnectionConfig<'d>\n    where\n        's: 'd,\n    {\n        ConnectionConfig {\n            database: Some(database),\n            ..*self\n        }\n    }\n\n    /// get a config string we can use to connect to the db\n    pub fn config_string(&self) -> String {\n        use std::fmt::Write;\n\n        let ConnectionConfig {\n            host,\n            port,\n            user,\n            password,\n            database,\n        } = self;\n\n        let mut config = String::new();\n        if let Some(host) = host {\n            let _ = write!(&mut config, \"host={host} \");\n        }\n        if let Some(port) = port {\n            let _ = write!(&mut config, \"port={port} \");\n        }\n        let _ = match user {\n            Some(user) => write!(&mut config, \"user={user} \"),\n            None => write!(&mut config, \"user=postgres \"),\n        };\n        if let Some(password) = password {\n            let _ = write!(&mut config, \"password={password} \");\n        }\n        if let Some(database) = database {\n            let _ = write!(&mut config, \"dbname={database} \");\n        }\n        config\n    }\n}\n"
  },
  {
    "path": "crates/stats-agg/Cargo.toml",
    "content": "[package]\nname = \"stats_agg\"\nversion = \"0.1.0\"\nauthors = [\"davidkohn88 <david@timescale.com>\"]\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nflat_serialize = {path=\"../flat_serialize/flat_serialize\"}\nflat_serialize_macro = {path=\"../flat_serialize/flat_serialize_macro\"}\nserde = { version = \"1.0\", features = [\"derive\"] }\ntwofloat = { version = \"0.8.4\", features = [\"serde\"] }\nnum-traits = \"0.2.15\"\n\n[dev-dependencies]\napprox = \"0.5.1\""
  },
  {
    "path": "crates/stats-agg/src/lib.rs",
    "content": "// stats is a small statistical regression lib that implements the Youngs-Cramer algorithm and is based on the Postgres implementation\n// here for 1D regression analysis:\n\n// And here for 2D regression analysis:\n// https://github.com/postgres/postgres/blob/472e518a44eacd9caac7d618f1b6451672ca4481/src/backend/utils/adt/float.c#L3260\n//\n\npub use twofloat::TwoFloat;\n\npub trait FloatLike:\n    num_traits::NumOps + num_traits::NumAssignOps + num_traits::Float + From<f64>\n{\n    /// Shorthand for `<T as From<f64>>::from(val)`\n    fn lit(val: f64) -> Self {\n        <Self as From<f64>>::from(val)\n    }\n    fn from_u64(n: u64) -> Self;\n}\nimpl FloatLike for f64 {\n    fn from_u64(n: u64) -> Self {\n        n as f64\n    }\n}\nimpl FloatLike for TwoFloat {\n    fn from_u64(n: u64) -> Self {\n        (n as f64).into()\n    }\n}\n\n#[derive(Debug, PartialEq, Eq)]\npub enum StatsError {\n    DoubleOverflow,\n}\n\n#[derive(Debug, PartialEq, Eq)]\npub struct XYPair<T: FloatLike> {\n    pub x: T,\n    pub y: T,\n}\n\n// The threshold at which we should re-calculate when we're doing the inverse transition in a windowed aggregate\n// essentially, if we're shifting the data by enough as we remove a value from the aggregate we can end up with\n// extra floating point error because in real arithmetic x = x + C - C\n// but in floating point arithmetic, if C is large compared to x, we can accumulate significant error.\n// In our case, because C is added in the normal transition or combine function, and then removed later in the\n// inverse function, we have x + C and C and we are testing the following: C / (x + C) > INV_FLOATING_ERROR_THRESHOLD\n// Because of the way that Postgres performs inverse functions, if we return a NULL value, the only thing that happens\n// is that the partial will get re-calculated from scratch from the values in the window function. So providing\n// the inverse function is purely an optimization. There are several cases where the C/(x + C) is likely to be larger\n// than our threshold, but we don't care too much, namely when there are one or two values this can happen frequently,\n// but then the cost of recalculation is low, compared to when there are many values in a rolling calculation, so we\n// test early in the function for whether we need to recalculate and pass NULL quickly so that we don't affect those\n// cases too heavily.\n#[cfg(not(test))]\nconst INV_FLOATING_ERROR_THRESHOLD: f64 = 0.99;\n#[cfg(test)] // don't have a threshold for tests, to ensure the inverse function is better tested\nconst INV_FLOATING_ERROR_THRESHOLD: f64 = f64::INFINITY;\n\npub mod stats1d;\npub mod stats2d;\n\n// This will wrap the logic for incrementing the sum for the third moment of a series of floats (i.e. Sum (i=1..N) of (i-avg)^3)\n// Math is sourced from https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Higher-order_statistics\nmod m3 {\n    use super::*;\n\n    // Add a value x to the set.  n, sx, sxx, sx3 are the values from prior to including x.\n    pub(crate) fn accum<T: FloatLike>(n: T, sx: T, sxx: T, sx3: T, x: T) -> T {\n        let delta = x - (sx / n);\n        let n = n + T::one();\n        sx3 + delta.powi(3) * (n - T::one()) * (n - T::lit(2.)) / n.powi(2)\n            - (T::lit(3.) * delta * sxx / n)\n    }\n    // Remove a value x from the set.  Here n, sx, sxx are all the values from the set after x has been removed.\n    // old_sx3 is the current value prior to the remove (sx3 after the removal is the returned value)\n    pub(crate) fn remove<T: FloatLike>(new_n: T, new_sx: T, new_sxx: T, old_sx3: T, x: T) -> T {\n        let delta = x - (new_sx / new_n);\n        let n = new_n + T::one();\n        old_sx3\n            - (delta.powi(3) * (n - T::one()) * (n - T::lit(2.)) / n.powi(2)\n                - (T::lit(3.) * delta * new_sxx / n))\n    }\n    // Combine two sets a and b and returns the sx3 for the combined set.\n    #[allow(clippy::too_many_arguments)]\n    pub(crate) fn combine<T: FloatLike>(\n        na: T,\n        nb: T,\n        sxa: T,\n        sxb: T,\n        sxxa: T,\n        sxxb: T,\n        sx3a: T,\n        sx3b: T,\n    ) -> T {\n        let nx = na + nb;\n        let delta = sxb / nb - sxa / na;\n        sx3a + sx3b\n            + delta.powi(3) * na * nb * (na - nb) / nx.powi(2)\n            + (na * sxxb - (nb * sxxa)) * T::lit(3.) * delta / nx\n    }\n    // This removes set b from a combined set, returning the sx3 of the remaining set a.\n    // Note that na, sxa, sxxa are all the values computed on the remaining set.  old_sx3 is the sx3 of the combined set.\n    #[allow(clippy::too_many_arguments)]\n    pub(crate) fn remove_combined<T: FloatLike>(\n        new_na: T,\n        nb: T,\n        new_sxa: T,\n        sxb: T,\n        new_sxxa: T,\n        sxxb: T,\n        old_sx3: T,\n        sx3b: T,\n    ) -> T {\n        let nx = new_na + nb;\n        let delta = sxb / nb - new_sxa / new_na;\n        old_sx3\n            - (sx3b\n                + delta.powi(3) * new_na * nb * (new_na - nb) / nx.powi(2)\n                + T::lit(3.) * (new_na * sxxb - (nb * new_sxxa)) * delta / nx)\n    }\n}\n\n// This will wrap the logic for incrementing the sum for the fourth moment of a series of floats (i.e. Sum (i=1..N) of (i-avg)^4)\n// Math is sourced from https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Higher-order_statistics\nmod m4 {\n    use super::*;\n\n    // Add a value x to the set.  n, sx, sxx, sx3, sx4 are the values from prior to including x.\n    pub(crate) fn accum<T: FloatLike>(n: T, sx: T, sxx: T, sx3: T, sx4: T, x: T) -> T {\n        let delta = x - (sx / n);\n        let n = n + T::one();\n        sx4 + delta.powi(4) * (n - T::one()) * (n.powi(2) - T::lit(3.) * n + T::lit(3.)) / n.powi(3)\n            + T::lit(6.) * delta.powi(2) * sxx / n.powi(2)\n            - T::lit(4.) * delta * sx3 / n\n    }\n    // Remove a value x from the set.  Here n, sx, sxx, sx3 are all the values from the set after x has been removed.\n    // old_sx4 is the current value prior to the remove (sx4 after the removal is the returned value)\n    pub(crate) fn remove<T: FloatLike>(\n        new_n: T,\n        new_sx: T,\n        new_sxx: T,\n        new_sx3: T,\n        old_sx4: T,\n        x: T,\n    ) -> T {\n        let delta = x - (new_sx / new_n);\n        let n = new_n + T::one();\n        old_sx4\n            - (delta.powi(4) * (n - T::one()) * (n.powi(2) - T::lit(3.) * n + T::lit(3.))\n                / n.powi(3)\n                + T::lit(6.) * delta.powi(2) * new_sxx / n.powi(2)\n                - T::lit(4.) * delta * new_sx3 / n)\n    }\n    // Combine two sets a and b and returns the sx4 for the combined set.\n    #[allow(clippy::too_many_arguments)]\n    pub(crate) fn combine<T: FloatLike>(\n        na: T,\n        nb: T,\n        sxa: T,\n        sxb: T,\n        sxxa: T,\n        sxxb: T,\n        sx3a: T,\n        sx3b: T,\n        sx4a: T,\n        sx4b: T,\n    ) -> T {\n        let nx = na + nb;\n        let delta = sxb / nb - sxa / na;\n        sx4a + sx4b\n            + delta.powi(4) * na * nb * (na.powi(2) - na * nb + nb.powi(2)) / nx.powi(3)\n            + T::lit(6.) * (na.powi(2) * sxxb + nb.powi(2) * sxxa) * delta.powi(2) / nx.powi(2)\n            + T::lit(4.) * (na * sx3b - nb * sx3a) * delta / nx\n    }\n    // This removes set b from a combined set, returning the sx4 of the remaining set a.\n    // Note that na, sxa, sxxa, sx3a are all the values computed on the remaining set.  old_sx4 is the sx4 of the combined set.\n    #[allow(clippy::too_many_arguments)]\n    pub(crate) fn remove_combined<T: FloatLike>(\n        new_na: T,\n        nb: T,\n        new_sxa: T,\n        sxb: T,\n        new_sxxa: T,\n        sxxb: T,\n        new_sx3a: T,\n        sx3b: T,\n        old_sx4: T,\n        sx4b: T,\n    ) -> T {\n        let nx = new_na + nb;\n        let delta = sxb / nb - new_sxa / new_na;\n        old_sx4\n            - (sx4b\n                + delta.powi(4) * new_na * nb * (new_na.powi(2) - new_na * nb + nb.powi(2))\n                    / nx.powi(3)\n                + T::lit(6.) * (new_na.powi(2) * sxxb + nb.powi(2) * new_sxxa) * delta.powi(2)\n                    / nx.powi(2)\n                + T::lit(4.) * (new_na * sx3b - nb * new_sx3a) * delta / nx)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use twofloat::TwoFloat;\n\n    #[test]\n    fn floatlike_lit() {\n        assert_eq!(f64::lit(3.), 3.);\n        assert_eq!(TwoFloat::lit(3.), TwoFloat::new_add(3., 0.));\n    }\n}\n"
  },
  {
    "path": "crates/stats-agg/src/stats1d.rs",
    "content": "use crate::{m3, m4, FloatLike, StatsError, TwoFloat, INV_FLOATING_ERROR_THRESHOLD};\nuse serde::{Deserialize, Serialize};\n\n#[derive(Debug, PartialEq, Eq, Copy, Clone, Serialize, Deserialize)]\n#[repr(C)]\npub struct StatsSummary1D<T: FloatLike> {\n    pub n: u64,\n    pub sx: T,\n    pub sx2: T,\n    pub sx3: T,\n    pub sx4: T,\n}\nimpl<T> Default for StatsSummary1D<T>\nwhere\n    T: FloatLike,\n{\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\n// can't make this impl generic without conflicting with the stdlib implementation of From<T> for T\nimpl From<StatsSummary1D<f64>> for StatsSummary1D<TwoFloat> {\n    fn from(input_summary: StatsSummary1D<f64>) -> Self {\n        StatsSummary1D {\n            n: input_summary.n,\n            sx: input_summary.sx.into(),\n            sx2: input_summary.sx2.into(),\n            sx3: input_summary.sx3.into(),\n            sx4: input_summary.sx4.into(),\n        }\n    }\n}\npub fn convert_tf_to_f64(tf: TwoFloat) -> f64 {\n    tf.hi() + tf.lo()\n}\nimpl From<StatsSummary1D<TwoFloat>> for StatsSummary1D<f64> {\n    fn from(input_summary: StatsSummary1D<TwoFloat>) -> Self {\n        StatsSummary1D {\n            n: input_summary.n,\n            sx: input_summary.sx.into(),\n            sx2: input_summary.sx2.into(),\n            sx3: input_summary.sx3.into(),\n            sx4: input_summary.sx4.into(),\n        }\n    }\n}\n\nimpl<T> StatsSummary1D<T>\nwhere\n    T: FloatLike,\n{\n    fn n64(&self) -> T {\n        T::from_u64(self.n)\n    }\n\n    pub fn new() -> Self {\n        StatsSummary1D {\n            n: 0,\n            sx: T::zero(),\n            sx2: T::zero(),\n            sx3: T::zero(),\n            sx4: T::zero(),\n        }\n    }\n\n    // we use the Youngs-Cramer method for accumulating the values here to allow for easy computation of variance etc in a numerically robust way.\n    // for this part, we've essentially copied the Postgres implementation found: // https://github.com/postgres/postgres/blob/8bdd6f563aa2456de602e78991e6a9f61b8ec86d/src/backend/utils/adt/float.c#L2813\n    // Note that the Youngs-Cramer method relies on the sum((x - Sx/n)^2) for which they derive a recurrence relation which is reflected in the algorithm here:\n    // the recurrence relation is: sum((x - Sx/n)^2) = Sxx = Sxx_n-1 + 1/(n(n-1)) * (nx - Sx)^2\n    pub fn accum(&mut self, p: T) -> Result<(), StatsError> {\n        let old = *self;\n        self.n += 1;\n        self.sx += p;\n        if old.n > 0 {\n            let tmpx = p * self.n64() - self.sx;\n            let scale = T::one() / (self.n64() * old.n64());\n            self.sx2 += tmpx * tmpx * scale;\n            self.sx3 = m3::accum(old.n64(), old.sx, old.sx2, old.sx3, p);\n            self.sx4 = m4::accum(old.n64(), old.sx, old.sx2, old.sx3, old.sx4, p);\n\n            if self.has_infinite() {\n                if self.check_overflow(&old, p) {\n                    return Err(StatsError::DoubleOverflow);\n                }\n                // sxx should be set to NaN if any of its inputs are\n                // infinite, so if they ended up as infinite and there wasn't an overflow,\n                // we need to set them to NaN instead as this implies that there was an\n                // infinite input (because they necessarily involve multiplications of\n                // infinites, which are NaNs)\n                if self.sx2.is_infinite() {\n                    self.sx2 = T::nan();\n                }\n                if self.sx3.is_infinite() {\n                    self.sx3 = T::nan();\n                }\n                if self.sx4.is_infinite() {\n                    self.sx4 = T::nan();\n                }\n            }\n        } else {\n            // first input, leave sxx alone unless we have infinite inputs\n            if !p.is_finite() {\n                self.sx2 = T::nan();\n                self.sx3 = T::nan();\n                self.sx4 = T::nan();\n            }\n        }\n        Result::Ok(())\n    }\n\n    fn has_infinite(&self) -> bool {\n        self.sx.is_infinite()\n            || self.sx2.is_infinite()\n            || self.sx3.is_infinite()\n            || self.sx4.is_infinite()\n    }\n\n    fn check_overflow(&self, old: &Self, p: T) -> bool {\n        //Only report overflow if we have finite inputs that lead to infinite results.\n        self.has_infinite() && old.sx.is_finite() && p.is_finite()\n    }\n\n    // inverse transition function (inverse of accum) for windowed aggregates, return None if we want to re-calculate from scratch\n    // we won't modify in place here because of that return bit, it might be that we want to modify accum to also\n    // copy just for symmetry.\n    // Assumption: no need for Result/error possibility because we can't overflow, as we are doing an inverse operation of something that already happened, so if it worked forward, it should work in reverse?\n    // We're extending the Youngs Cramer algorithm here with the algebraic transformation to figure out the reverse calculations.\n    // This goes beyond what the PG code does, and is our extension for performance in windowed calculations.\n\n    // There is a case where the numerical error can get very large that we will try to avoid: if we have an outlier value that is much larger than the surrounding values\n    // we can get something like: v1 + v2 + v3 + ... vn = outlier + v1 + v2 + v3 + ... + vn - outlier when the outlier is removed from the window. This will cause significant error in the\n    // resulting calculation of v1 + ... + vn, more than we're comfortable with, so we'll return None in that case which will force recalculation from scratch of v1 + ... + vn.\n\n    // Algebra for removal:\n    // n = n_old + 1 -> n_old = n - 1\n    // Sx = Sx_old + x -> Sx_old = Sx - x\n    // sum((x - Sx/n)^2) = Sxx = Sxx_old + 1/(n * n_old) * (nx - Sx)^2  -> Sxx_old = Sxx - 1/(n * n_old) * (nx - Sx)^2\n\n    pub fn remove(&self, p: T) -> Option<Self> {\n        // if we are trying to remove a nan/infinite input, it's time to recalculate.\n        if p.is_nan() || p.is_infinite() {\n            return None;\n        }\n        // if we are removing a value that is very large compared to the sum of the values that we're removing it from,\n        // we should probably recalculate to avoid accumulating error. We might want a different test for this, if there\n        // is a  way to calculate the error directly, that might be best...\n        if p / self.sx > <T as From<f64>>::from(INV_FLOATING_ERROR_THRESHOLD) {\n            return None;\n        }\n\n        // we can't have an initial value of n = 0 if we're removing something...\n        if self.n == 0 {\n            panic!(); //perhaps we should do error handling here, but I think this is reasonable as we are assuming that the removal is of an already-added item in the rest of this\n        }\n\n        if self.n == 1 {\n            return Some(StatsSummary1D::new());\n        }\n\n        let mut new = StatsSummary1D {\n            n: self.n - 1,\n            sx: self.sx - p,\n            sx2: T::zero(), // initialize this for now.\n            sx3: T::zero(), // initialize this for now.\n            sx4: T::zero(), // initialize this for now.\n        };\n\n        let tmpx = p * self.n64() - self.sx;\n        let scale = (self.n64() * new.n64()).recip();\n        new.sx2 = self.sx2 - tmpx * tmpx * scale;\n        new.sx3 = m3::remove(new.n64(), new.sx, new.sx2, self.sx3, p);\n        new.sx4 = m4::remove(new.n64(), new.sx, new.sx2, new.sx3, self.sx4, p);\n\n        Some(new)\n    }\n\n    // convenience function for creating an aggregate from a vector, currently used mostly for testing.\n    pub fn new_from_vec(v: Vec<T>) -> Result<Self, StatsError> {\n        let mut r = StatsSummary1D::new();\n        for p in v {\n            r.accum(p)?;\n        }\n        Result::Ok(r)\n    }\n\n    pub fn combine(&self, other: Self) -> Result<Self, StatsError> {\n        // TODO: think about whether we want to just modify &self in place here for perf\n        // reasons. This is also a set of weird questions around the Rust compiler, so\n        // easier to just add the copy trait here, may need to adjust or may make things\n        // harder if we do generics.\n        if self.n == 0 && other.n == 0 {\n            return Ok(StatsSummary1D::new());\n        } else if self.n == 0 {\n            // handle the trivial n = 0 cases here, and don't worry about divide by zero errors later.\n            return Ok(other);\n        } else if other.n == 0 {\n            return Ok(*self);\n        }\n        let tmp = self.sx / self.n64() - other.sx / other.n64();\n        let n = self.n + other.n;\n        let r = StatsSummary1D {\n            n,\n            sx: self.sx + other.sx,\n            sx2: self.sx2\n                + other.sx2\n                + self.n64() * other.n64() * tmp * tmp\n                    / <T as num_traits::cast::NumCast>::from(n).unwrap(),\n            sx3: m3::combine(\n                self.n64(),\n                other.n64(),\n                self.sx,\n                other.sx,\n                self.sx2,\n                other.sx2,\n                self.sx3,\n                other.sx3,\n            ),\n            sx4: m4::combine(\n                self.n64(),\n                other.n64(),\n                self.sx,\n                other.sx,\n                self.sx2,\n                other.sx2,\n                self.sx3,\n                other.sx3,\n                self.sx4,\n                other.sx4,\n            ),\n        };\n        if r.has_infinite() && !self.has_infinite() && !other.has_infinite() {\n            return Err(StatsError::DoubleOverflow);\n        }\n        Ok(r)\n    }\n\n    // This is the inverse combine function for use in the window function context when we want to reverse the operation of the normal combine function\n    // for re-aggregation over a window, this is what will get called in tumbling window averages for instance.\n    // As with any window function, returning None will cause a re-calculation, so we do that in several cases where either we're dealing with infinites or we have some potential problems with outlying sums\n    // so here, self is the previously combined StatsSummary, and we're removing the input and returning the part that would have been there before.\n    pub fn remove_combined(&self, remove: Self) -> Option<Self> {\n        let combined = &self; // just to lessen confusion with naming\n                              // handle the trivial n = 0 and equal n cases here, and don't worry about divide by zero errors later.\n        if combined.n == remove.n {\n            return Some(StatsSummary1D::new());\n        } else if remove.n == 0 {\n            return Some(*self);\n        } else if combined.n < remove.n {\n            panic!(); // given that we're always removing things that we've previously added, we shouldn't be able to get a case where we're removing an n that's larger.\n        }\n        // if the sum we're removing is very large compared to the overall value we need to recalculate, see note on the remove function\n        if remove.sx / combined.sx > <T as From<f64>>::from(INV_FLOATING_ERROR_THRESHOLD) {\n            return None;\n        }\n        let mut part = StatsSummary1D {\n            n: combined.n - remove.n,\n            sx: combined.sx - remove.sx,\n            sx2: T::zero(), //just initialize this, for now.\n            sx3: T::zero(), //just initialize this, for now.\n            sx4: T::zero(), //just initialize this, for now.\n        };\n        let tmp = part.sx / part.n64() - remove.sx / remove.n64(); //gets squared so order doesn't matter\n        part.sx2 =\n            combined.sx2 - remove.sx2 - part.n64() * remove.n64() * tmp * tmp / combined.n64();\n        part.sx3 = m3::remove_combined(\n            part.n64(),\n            remove.n64(),\n            part.sx,\n            remove.sx,\n            part.sx2,\n            remove.sx2,\n            self.sx3,\n            remove.sx3,\n        );\n        part.sx4 = m4::remove_combined(\n            part.n64(),\n            remove.n64(),\n            part.sx,\n            remove.sx,\n            part.sx2,\n            remove.sx2,\n            part.sx3,\n            remove.sx3,\n            self.sx4,\n            remove.sx4,\n        );\n\n        Some(part)\n    }\n\n    pub fn avg(&self) -> Option<T> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(self.sx / self.n64())\n    }\n\n    pub fn count(&self) -> i64 {\n        self.n as i64\n    }\n\n    pub fn sum(&self) -> Option<T> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(self.sx)\n    }\n\n    pub fn var_pop(&self) -> Option<T> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(self.sx2 / self.n64())\n    }\n\n    pub fn var_samp(&self) -> Option<T> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(self.sx2 / (self.n64() - T::one()))\n    }\n\n    pub fn stddev_pop(&self) -> Option<T> {\n        Some(self.var_pop()?.sqrt())\n    }\n\n    pub fn stddev_samp(&self) -> Option<T> {\n        Some(self.var_samp()?.sqrt())\n    }\n\n    pub fn skewness_pop(&self) -> Option<T> {\n        Some(self.sx3 / self.n64() / self.stddev_pop()?.powi(3))\n    }\n\n    pub fn skewness_samp(&self) -> Option<T> {\n        Some(self.sx3 / (self.n64() - T::one()) / self.stddev_samp()?.powi(3))\n    }\n\n    pub fn kurtosis_pop(&self) -> Option<T> {\n        Some(self.sx4 / self.n64() / self.stddev_pop()?.powi(4))\n    }\n\n    pub fn kurtosis_samp(&self) -> Option<T> {\n        Some(self.sx4 / (self.n64() - T::one()) / self.stddev_samp()?.powi(4))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use approx::assert_relative_eq;\n\n    fn tf(f: f64) -> TwoFloat {\n        TwoFloat::new_add(f, 0.0)\n    }\n\n    #[track_caller]\n    fn assert_close_enough(s1: &StatsSummary1D<f64>, s2: &StatsSummary1D<f64>) {\n        assert_eq!(s1.n, s2.n);\n        assert_relative_eq!(s1.sx, s2.sx);\n        assert_relative_eq!(s1.sx2, s2.sx2);\n        assert_relative_eq!(s1.sx3, s2.sx3);\n        assert_relative_eq!(s1.sx4, s2.sx4);\n    }\n\n    #[track_caller]\n    fn assert_close_enough_tf(s1: &StatsSummary1D<TwoFloat>, s2: &StatsSummary1D<TwoFloat>) {\n        assert_eq!(s1.n, s2.n);\n        assert!((s1.sx - s2.sx).abs() < 10.0 * f64::EPSILON);\n        assert!((s1.sx2 - s2.sx2).abs() < 10.0 * f64::EPSILON);\n        assert!((s1.sx3 - s2.sx3).abs() < 10.0 * f64::EPSILON);\n        assert!((s1.sx4 - s2.sx4).abs() < 10.0 * f64::EPSILON);\n    }\n\n    #[test]\n    fn test_against_known_vals() {\n        let p = StatsSummary1D::new_from_vec(vec![7.0, 18.0, -2.0, 5.0, 3.0]).unwrap();\n\n        assert_eq!(p.n, 5);\n        assert_relative_eq!(p.sx, 31.);\n        assert_relative_eq!(p.sx2, 218.8);\n        assert_relative_eq!(p.sx3, 1057.68);\n        assert_relative_eq!(p.sx4, 24016.336);\n\n        let p = p.remove(18.0).unwrap();\n\n        assert_eq!(p.n, 4);\n        assert_relative_eq!(p.sx, 13.);\n        assert_relative_eq!(p.sx2, 44.75);\n        assert_relative_eq!(p.sx3, -86.625);\n        assert_relative_eq!(p.sx4, 966.8281249999964);\n\n        let p = p\n            .combine(StatsSummary1D::new_from_vec(vec![0.5, 11.0, 6.123]).unwrap())\n            .unwrap();\n\n        assert_eq!(p.n, 7);\n        assert_relative_eq!(p.sx, 30.623);\n        assert_relative_eq!(p.sx2, 111.77425342857143);\n        assert_relative_eq!(p.sx3, -5.324891254897949);\n        assert_relative_eq!(p.sx4, 3864.054085451184);\n\n        let p = p\n            .remove_combined(StatsSummary1D::new_from_vec(vec![5.0, 11.0, 3.0]).unwrap())\n            .unwrap();\n\n        assert_eq!(p.n, 4);\n        assert_relative_eq!(p.sx, 11.623);\n        assert_relative_eq!(p.sx2, 56.96759675000001);\n        assert_relative_eq!(p.sx3, -30.055041237374915);\n        assert_relative_eq!(p.sx4, 1000.8186787745212);\n    }\n\n    #[test]\n    fn test_against_known_vals_tf() {\n        let p = StatsSummary1D::new_from_vec(vec![tf(7.0), tf(18.0), tf(-2.0), tf(5.0), tf(3.0)])\n            .unwrap();\n\n        assert_eq!(p.n, 5);\n        assert_relative_eq!(Into::<f64>::into(p.sx), 31.);\n        assert_relative_eq!(Into::<f64>::into(p.sx2), 218.8);\n        assert_relative_eq!(Into::<f64>::into(p.sx3), 1057.68);\n        assert_relative_eq!(Into::<f64>::into(p.sx4), 24016.336);\n\n        let p = p.remove(tf(18.0)).unwrap();\n\n        assert_eq!(p.n, 4);\n        assert_relative_eq!(Into::<f64>::into(p.sx), 13.);\n        // value is slightly off\n        assert_relative_eq!(Into::<f64>::into(p.sx2), 44.75, epsilon = 0.000000000001);\n        assert_relative_eq!(Into::<f64>::into(p.sx3), -86.625, epsilon = 0.000000000001);\n        assert_relative_eq!(\n            Into::<f64>::into(p.sx4),\n            966.8281249999964,\n            epsilon = 0.000000000001\n        );\n\n        let p = p\n            .combine(StatsSummary1D::new_from_vec(vec![tf(0.5), tf(11.0), tf(6.123)]).unwrap())\n            .unwrap();\n\n        assert_eq!(p.n, 7);\n        assert_relative_eq!(Into::<f64>::into(p.sx), 30.623);\n        assert_relative_eq!(Into::<f64>::into(p.sx2), 111.77425342857143);\n        // slight difference in values here – not sure if twofloat or f64 is more accurate\n        assert_relative_eq!(\n            Into::<f64>::into(p.sx3),\n            -5.324891254897949,\n            epsilon = 0.0000000001\n        );\n        assert_relative_eq!(\n            Into::<f64>::into(p.sx4),\n            3864.054085451184,\n            epsilon = 0.0000000001\n        );\n\n        let p = p\n            .remove_combined(\n                StatsSummary1D::new_from_vec(vec![tf(5.0), tf(11.0), tf(3.0)]).unwrap(),\n            )\n            .unwrap();\n\n        assert_eq!(p.n, 4);\n        assert_relative_eq!(Into::<f64>::into(p.sx), 11.623);\n        // f64 gets this slightly over, TF gets this slightly under\n        assert_relative_eq!(\n            Into::<f64>::into(p.sx2),\n            56.96759675000001,\n            epsilon = 0.000000000001\n        );\n        // slight difference in values here – not sure if twofloat or f64 is more accurate\n        assert_relative_eq!(\n            Into::<f64>::into(p.sx3),\n            -30.055041237374915,\n            epsilon = 0.0000000001\n        );\n        assert_relative_eq!(\n            Into::<f64>::into(p.sx4),\n            1000.8186787745212,\n            epsilon = 0.0000000001\n        );\n    }\n\n    #[test]\n    fn test_combine() {\n        let p = StatsSummary1D::new_from_vec(vec![1.0, 2.0, 3.0, 4.0]).unwrap();\n        let q = StatsSummary1D::new_from_vec(vec![1.0, 2.0]).unwrap();\n        let r = StatsSummary1D::new_from_vec(vec![3.0, 4.0]).unwrap();\n        assert_close_enough(&q.combine(r).unwrap(), &p);\n\n        let p = StatsSummary1D::new_from_vec(vec![tf(1.0), tf(2.0), tf(3.0), tf(4.0)]).unwrap();\n        let q = StatsSummary1D::new_from_vec(vec![tf(1.0), tf(2.0)]).unwrap();\n        let r = StatsSummary1D::new_from_vec(vec![tf(3.0), tf(4.0)]).unwrap();\n        assert_close_enough_tf(&q.combine(r).unwrap(), &p);\n    }\n}\n"
  },
  {
    "path": "crates/stats-agg/src/stats2d/stats2d_flat_serialize.rs",
    "content": "use super::*;\n\n// expanded from FlatSerializable derive macro and made to work right with generic arg\n#[allow(warnings, clippy::all)]\nunsafe impl<'a> flat_serialize::FlatSerializable<'a> for StatsSummary2D<f64> {\n    const REQUIRED_ALIGNMENT: usize = {\n        use std::mem::align_of;\n        let mut required_alignment = 1;\n        let alignment = <u64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        let alignment = <f64 as flat_serialize::FlatSerializable>::REQUIRED_ALIGNMENT;\n        if alignment > required_alignment {\n            required_alignment = alignment;\n        }\n        required_alignment\n    };\n    const MAX_PROVIDED_ALIGNMENT: Option<usize> = {\n        use std::mem::align_of;\n        let mut min_align: Option<usize> = None;\n        let ty_align = <u64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        let ty_align = <f64 as flat_serialize::FlatSerializable>::MAX_PROVIDED_ALIGNMENT;\n        match (ty_align, min_align) {\n            (None, _) => {}\n            (Some(align), None) => min_align = Some(align),\n            (Some(align), Some(min)) if align < min => min_align = Some(align),\n            _ => {}\n        }\n        match min_align {\n            None => None,\n            Some(min_align) => {\n                let min_size = Self::MIN_LEN;\n                if min_size % 8 == 0 && min_align >= 8 {\n                    Some(8)\n                } else if min_size % 4 == 0 && min_align >= 4 {\n                    Some(4)\n                } else if min_size % 2 == 0 && min_align >= 2 {\n                    Some(2)\n                } else {\n                    Some(1)\n                }\n            }\n        }\n    };\n    const MIN_LEN: usize = {\n        use std::mem::size_of;\n        let mut size = 0;\n        size += <u64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size += <f64 as flat_serialize::FlatSerializable>::MIN_LEN;\n        size\n    };\n    const TRIVIAL_COPY: bool = true;\n    type SLICE = flat_serialize::Slice<'a, StatsSummary2D<f64>>;\n    type OWNED = Self;\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn try_ref(mut input: &[u8]) -> Result<(Self, &[u8]), flat_serialize::WrapErr> {\n        if input.len() < Self::MIN_LEN {\n            return Err(flat_serialize::WrapErr::NotEnoughBytes(Self::MIN_LEN));\n        }\n        let __packet_macro_read_len = 0usize;\n        let mut n: Option<u64> = None;\n        let mut sx: Option<f64> = None;\n        let mut sx2: Option<f64> = None;\n        let mut sx3: Option<f64> = None;\n        let mut sx4: Option<f64> = None;\n        let mut sy: Option<f64> = None;\n        let mut sy2: Option<f64> = None;\n        let mut sy3: Option<f64> = None;\n        let mut sy4: Option<f64> = None;\n        let mut sxy: Option<f64> = None;\n        'tryref: loop {\n            {\n                let (field, rem) = match <u64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                n = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sx = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sx2 = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sx3 = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sx4 = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sy = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sy2 = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sy3 = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sy4 = Some(field);\n            }\n            {\n                let (field, rem) = match <f64>::try_ref(input) {\n                    Ok((f, b)) => (f, b),\n                    Err(flat_serialize::WrapErr::InvalidTag(offset)) => {\n                        return Err(flat_serialize::WrapErr::InvalidTag(\n                            __packet_macro_read_len + offset,\n                        ));\n                    }\n                    Err(..) => break 'tryref,\n                };\n                input = rem;\n                sxy = Some(field);\n            }\n            let _ref = StatsSummary2D {\n                n: n.unwrap(),\n                sx: sx.unwrap(),\n                sx2: sx2.unwrap(),\n                sx3: sx3.unwrap(),\n                sx4: sx4.unwrap(),\n                sy: sy.unwrap(),\n                sy2: sy2.unwrap(),\n                sy3: sy3.unwrap(),\n                sy4: sy4.unwrap(),\n                sxy: sxy.unwrap(),\n            };\n            return Ok((_ref, input));\n        }\n        Err(flat_serialize::WrapErr::NotEnoughBytes(\n            0 + <u64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN\n                + <f64>::MIN_LEN,\n        ))\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    unsafe fn fill_slice<'out>(\n        &self,\n        input: &'out mut [std::mem::MaybeUninit<u8>],\n    ) -> &'out mut [std::mem::MaybeUninit<u8>] {\n        let total_len = self.num_bytes();\n        let (mut input, rem) = input.split_at_mut(total_len);\n        let StatsSummary2D {\n            n,\n            sx,\n            sx2,\n            sx3,\n            sx4,\n            sy,\n            sy2,\n            sy3,\n            sy4,\n            sxy,\n        } = self;\n        unsafe {\n            input = n.fill_slice(input);\n        };\n        unsafe {\n            input = sx.fill_slice(input);\n        };\n        unsafe {\n            input = sx2.fill_slice(input);\n        };\n        unsafe {\n            input = sx3.fill_slice(input);\n        };\n        unsafe {\n            input = sx4.fill_slice(input);\n        };\n        unsafe {\n            input = sy.fill_slice(input);\n        };\n        unsafe {\n            input = sy2.fill_slice(input);\n        };\n        unsafe {\n            input = sy3.fill_slice(input);\n        };\n        unsafe {\n            input = sy4.fill_slice(input);\n        };\n        unsafe {\n            input = sxy.fill_slice(input);\n        }\n        if true {\n            match (&input.len(), &0) {\n                (left_val, right_val) => {\n                    debug_assert_eq!(input.len(), 0);\n                }\n            };\n        }\n        rem\n    }\n    #[allow(unused_assignments, unused_variables)]\n    #[inline(always)]\n    fn num_bytes(&self) -> usize {\n        let StatsSummary2D {\n            n,\n            sx,\n            sx2,\n            sx3,\n            sx4,\n            sy,\n            sy2,\n            sy3,\n            sy4,\n            sxy,\n        } = self;\n        0usize\n            + <u64 as flat_serialize::FlatSerializable>::num_bytes(n)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sx)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sx2)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sx3)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sx4)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sy)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sy2)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sy3)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sy4)\n            + <f64 as flat_serialize::FlatSerializable>::num_bytes(sxy)\n    }\n    #[inline(always)]\n    fn make_owned(&mut self) {}\n    #[inline(always)]\n    fn into_owned(self) -> Self::OWNED {\n        self\n    }\n}\n"
  },
  {
    "path": "crates/stats-agg/src/stats2d.rs",
    "content": "// 2D stats are based on the Youngs-Cramer implementation in PG here:\n// https://github.com/postgres/postgres/blob/472e518a44eacd9caac7d618f1b6451672ca4481/src/backend/utils/adt/float.c#L3260\nuse crate::{m3, m4, FloatLike, StatsError, XYPair, INV_FLOATING_ERROR_THRESHOLD};\nuse serde::{Deserialize, Serialize};\nuse twofloat::TwoFloat;\n\nmod stats2d_flat_serialize;\n\n#[derive(Debug, PartialEq, Eq, Copy, Clone, Serialize, Deserialize)]\n#[repr(C)]\npub struct StatsSummary2D<T: FloatLike> {\n    pub n: u64, // count\n    pub sx: T,  // sum(x)\n    pub sx2: T, // sum((x-sx/n)^2) (sum of squares)\n    pub sx3: T, // sum((x-sx/n)^3)\n    pub sx4: T, // sum((x-sx/n)^4)\n    pub sy: T,  // sum(y)\n    pub sy2: T, // sum((y-sy/n)^2) (sum of squares)\n    pub sy3: T, // sum((y-sy/n)^3)\n    pub sy4: T, // sum((y-sy/n)^4)\n    pub sxy: T, // sum((x-sx/n)*(y-sy/n)) (sum of products)\n}\n\nimpl From<StatsSummary2D<TwoFloat>> for StatsSummary2D<f64> {\n    fn from(input_summary: StatsSummary2D<TwoFloat>) -> Self {\n        StatsSummary2D {\n            n: input_summary.n,\n            sx: input_summary.sx.into(),\n            sx2: input_summary.sx2.into(),\n            sx3: input_summary.sx3.into(),\n            sx4: input_summary.sx4.into(),\n            sy: input_summary.sy.into(),\n            sy2: input_summary.sy2.into(),\n            sy3: input_summary.sy3.into(),\n            sy4: input_summary.sy4.into(),\n            sxy: input_summary.sxy.into(),\n        }\n    }\n}\n\nimpl<T: FloatLike> Default for StatsSummary2D<T> {\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\nimpl<T: FloatLike> StatsSummary2D<T> {\n    pub fn new() -> Self {\n        StatsSummary2D {\n            n: 0,\n            sx: T::zero(),\n            sx2: T::zero(),\n            sx3: T::zero(),\n            sx4: T::zero(),\n            sy: T::zero(),\n            sy2: T::zero(),\n            sy3: T::zero(),\n            sy4: T::zero(),\n            sxy: T::zero(),\n        }\n    }\n\n    fn n64(&self) -> T {\n        T::from_u64(self.n)\n    }\n    /// accumulate an XYPair into a StatsSummary2D\n    /// ```\n    /// use stats_agg::*;\n    /// use stats_agg::stats2d::*;\n    /// let mut p = StatsSummary2D::new();\n    /// p.accum(XYPair{x:1.0, y:1.0,}).unwrap();\n    /// p.accum(XYPair{x:2.0, y:2.0,}).unwrap();\n    /// //we can add in infinite values and it will handle it properly.\n    /// p.accum(XYPair{x:f64::INFINITY, y:1.0}).unwrap();\n    /// assert_eq!(p.sum().unwrap().x, f64::INFINITY);\n    /// assert!(p.sum_squares().unwrap().x.is_nan()); // this is NaN because it involves multiplication of two infinite values\n    ///\n    /// assert_eq!(p.accum(XYPair{y:f64::MAX, x:1.0,}), Err(StatsError::DoubleOverflow)); // we do error if we actually overflow however\n    ///\n    ///```\n    pub fn accum(&mut self, p: XYPair<T>) -> Result<(), StatsError> {\n        let old = *self;\n        self.n += 1;\n        self.sx += p.x;\n        self.sy += p.y;\n        if old.n > 0 {\n            let tmpx = p.x * self.n64() - self.sx;\n            let tmpy = p.y * self.n64() - self.sy;\n            let scale = (self.n64() * old.n64()).recip();\n            self.sx2 += tmpx * tmpx * scale;\n            self.sx3 = m3::accum(old.n64(), old.sx, old.sx2, old.sx3, p.x);\n            self.sx4 = m4::accum(old.n64(), old.sx, old.sx2, old.sx3, old.sx4, p.x);\n            self.sy2 += tmpy * tmpy * scale;\n            self.sy3 = m3::accum(old.n64(), old.sy, old.sy2, old.sy3, p.y);\n            self.sy4 = m4::accum(old.n64(), old.sy, old.sy2, old.sy3, old.sy4, p.y);\n            self.sxy += tmpx * tmpy * scale;\n            if self.has_infinite() {\n                if self.check_overflow(&old, p) {\n                    return Err(StatsError::DoubleOverflow);\n                }\n                // sxx, syy, and sxy should be set to NaN if any of their inputs are\n                // infinite, so if they ended up as infinite and there wasn't an overflow,\n                // we need to set them to NaN instead as this implies that there was an\n                // infinite input (because they necessarily involve multiplications of\n                // infinites, which are NaNs)\n                if self.sx2.is_infinite() {\n                    self.sx2 = T::nan();\n                }\n                if self.sx3.is_infinite() {\n                    self.sx3 = T::nan();\n                }\n                if self.sx4.is_infinite() {\n                    self.sx4 = T::nan();\n                }\n                if self.sy2.is_infinite() {\n                    self.sy2 = T::nan();\n                }\n                if self.sy3.is_infinite() {\n                    self.sy3 = T::nan();\n                }\n                if self.sy4.is_infinite() {\n                    self.sy4 = T::nan();\n                }\n                if self.sxy.is_infinite() {\n                    self.sxy = T::nan();\n                }\n            }\n        } else {\n            // first input, leave sxx/syy/sxy alone unless we have infinite inputs\n            if !p.x.is_finite() {\n                self.sx2 = T::nan();\n                self.sx3 = T::nan();\n                self.sx4 = T::nan();\n                self.sxy = T::nan();\n            }\n            if !p.y.is_finite() {\n                self.sy2 = T::nan();\n                self.sy3 = T::nan();\n                self.sy4 = T::nan();\n                self.sxy = T::nan();\n            }\n        }\n        Result::Ok(())\n    }\n    fn has_infinite(&self) -> bool {\n        self.sx.is_infinite()\n            || self.sx2.is_infinite()\n            || self.sx3.is_infinite()\n            || self.sx4.is_infinite()\n            || self.sy.is_infinite()\n            || self.sy2.is_infinite()\n            || self.sy3.is_infinite()\n            || self.sy4.is_infinite()\n            || self.sxy.is_infinite()\n    }\n    fn check_overflow(&self, old: &StatsSummary2D<T>, p: XYPair<T>) -> bool {\n        //Only report overflow if we have finite inputs that lead to infinite results.\n        ((self.sx.is_infinite()\n            || self.sx2.is_infinite()\n            || self.sx3.is_infinite()\n            || self.sx4.is_infinite())\n            && old.sx.is_finite()\n            && p.x.is_finite())\n            || ((self.sy.is_infinite()\n                || self.sy2.is_infinite()\n                || self.sy3.is_infinite()\n                || self.sy4.is_infinite())\n                && old.sy.is_finite()\n                && p.y.is_finite())\n            || (self.sxy.is_infinite()\n                && old.sx.is_finite()\n                && p.x.is_finite()\n                && old.sy.is_finite()\n                && p.y.is_finite())\n    }\n\n    // inverse transition function (inverse of accum) for windowed aggregates, return None if we want to re-calculate from scratch\n    // we won't modify in place here because of that return bit, it might be that we want to modify accum to also\n    // copy just for symmetry.\n    // Assumption: no need for Result/error possibility because we can't overflow, as we are doing an inverse operation of something that already happened, so if it worked forward, it should work in reverse?\n    // We're extending the Youngs Cramer algorithm here with the algebraic transformation to figure out the reverse calculations.\n    // This goes beyond what the PG code does, and is our extension for performance in windowed calculations.\n\n    // There is a case where the numerical error can get very large that we will try to avoid: if we have an outlier value that is much larger than the surrounding values\n    // we can get something like: v1 + v2 + v3 + ... vn = outlier + v1 + v2 + v3 + ... + vn - outlier when the outlier is removed from the window. This will cause significant error in the\n    // resulting calculation of v1 + ... + vn, more than we're comfortable with, so we'll return None in that case which will force recalculation from scratch of v1 + ... + vn.\n\n    // Algebra for removal:\n    // n = n_old + 1 -> n_old = n - 1\n    // Sx = Sx_old + x -> Sx_old = Sx - x\n    // sum((x - Sx/n)^2) = Sxx = Sxx_old + 1/(n * n_old) * (nx - Sx)^2  -> Sxx_old = Sxx - 1/(n * n_old) * (nx - Sx)^2\n    // Sy / Syy analogous\n    // sum((x - Sx/n)(y - Sy/n)) = Sxy = Sxy_old + 1/(n * n_old) * (nx - Sx) * (ny - Sy)  -> Sxy_old = Sxy - 1/(n * n_old) * (nx - Sx) * (ny - Sy)\n    pub fn remove(&self, p: XYPair<T>) -> Option<Self> {\n        // if we are trying to remove a nan/infinite input, it's time to recalculate.\n        if !p.x.is_finite() || !p.y.is_finite() {\n            return None;\n        }\n        // if we are removing a value that is very large compared to the sum of the values that we're removing it from,\n        // we should probably recalculate to avoid accumulating error. We might want a different test for this, if there\n        // is a  way to calculate the error directly, that might be best...\n        let thresh = <T as From<f64>>::from(INV_FLOATING_ERROR_THRESHOLD);\n        if p.x / self.sx > thresh || p.y / self.sy > thresh {\n            return None;\n        }\n\n        // we can't have an initial value of n = 0 if we're removing something...\n        if self.n == 0 {\n            panic!(); //perhaps we should do error handling here, but I think this is reasonable as we are assuming that the removal is of an already-added item in the rest of this\n        }\n\n        // if we're removing the last point we should just return a completely empty value to eliminate any errors, it can only be completely empty at that point.\n        if self.n == 1 {\n            return Some(StatsSummary2D::new());\n        }\n\n        let mut new = StatsSummary2D {\n            n: self.n - 1,\n            sx: self.sx - p.x,\n            sy: self.sy - p.y,\n            sx2: T::zero(), // initialize these for now.\n            sx3: T::zero(),\n            sx4: T::zero(),\n            sy2: T::zero(),\n            sy3: T::zero(),\n            sy4: T::zero(),\n            sxy: T::zero(),\n        };\n        let tmpx = p.x * self.n64() - self.sx;\n        let tmpy = p.y * self.n64() - self.sy;\n        let scale = (self.n64() * new.n64()).recip();\n        new.sx2 = self.sx2 - tmpx * tmpx * scale;\n        new.sx3 = m3::remove(new.n64(), new.sx, new.sx2, self.sx3, p.x);\n        new.sx4 = m4::remove(new.n64(), new.sx, new.sx2, new.sx3, self.sx4, p.x);\n        new.sy2 = self.sy2 - tmpy * tmpy * scale;\n        new.sy3 = m3::remove(new.n64(), new.sy, new.sy2, self.sy3, p.y);\n        new.sy4 = m4::remove(new.n64(), new.sy, new.sy2, new.sy3, self.sy4, p.y);\n        new.sxy = self.sxy - tmpx * tmpy * scale;\n        Some(new)\n    }\n\n    ///create a StatsSummary2D from a vector of XYPairs\n    /// ```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let mut p = StatsSummary2D::new();\n    /// p.accum(XYPair{x:1.0, y:1.0,}).unwrap();\n    /// p.accum(XYPair{x:2.0, y:2.0,}).unwrap();\n    /// p.accum(XYPair{x:3.0, y:3.0,}).unwrap();\n    /// let q = StatsSummary2D::new_from_vec(vec![XYPair{x:1.0, y:1.0,}, XYPair{x:2.0, y:2.0,}, XYPair{x:3.0, y:3.0,}]).unwrap();\n    /// assert_eq!(p, q);\n    ///```\n    pub fn new_from_vec(v: Vec<XYPair<T>>) -> Result<Self, StatsError> {\n        let mut r = StatsSummary2D::new();\n        for p in v {\n            r.accum(p)?;\n        }\n        Result::Ok(r)\n    }\n    /// combine two StatsSummary2Ds\n    /// ```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{x:1.0, y:1.0,}, XYPair{x:2.0, y:2.0,}, XYPair{x:3.0, y:3.0,}, XYPair{x:4.0, y:4.0,}]).unwrap();\n    /// let q = StatsSummary2D::new_from_vec(vec![XYPair{x:1.0, y:1.0,}, XYPair{x:2.0, y:2.0,},]).unwrap();\n    /// let r = StatsSummary2D::new_from_vec(vec![XYPair{x:3.0, y:3.0,}, XYPair{x:4.0, y:4.0,},]).unwrap();\n    /// let r = r.combine(q).unwrap();\n    /// assert_eq!(r, p);\n    /// ```\n    // we combine two StatsSummary2Ds via a generalization of the Youngs-Cramer algorithm, we follow what Postgres does here\n    //      n = n1 + n2\n    //      sx = sx1 + sx2\n    //      sxx = sxx1 + sxx2 + n1 * n2 * (sx1/n1 - sx2/n2)^2 / n\n    //      sy / syy analogous\n    //      sxy = sxy1 + sxy2 + n1 * n2 * (sx1/n1 - sx2/n2) * (sy1/n1 - sy2/n2) / n\n    pub fn combine(&self, other: StatsSummary2D<T>) -> Result<Self, StatsError> {\n        // TODO: think about whether we want to just modify &self in place here for perf\n        // reasons. This is also a set of weird questions around the Rust compiler, so\n        // easier to just add the copy trait here, may need to adjust or may make things\n        // harder if we do generics.\n        if self.n == 0 && other.n == 0 {\n            return Ok(StatsSummary2D::new());\n        } else if self.n == 0 {\n            // handle the trivial n = 0 cases here, and don't worry about divide by zero errors later.\n            return Ok(other);\n        } else if other.n == 0 {\n            return Ok(*self);\n        }\n        let tmpx = self.sx / self.n64() - other.sx / other.n64();\n        let tmpy = self.sy / self.n64() - other.sy / other.n64();\n        let n = self.n + other.n;\n        let r = StatsSummary2D {\n            n,\n            sx: self.sx + other.sx,\n            sx2: self.sx2 + other.sx2 + self.n64() * other.n64() * tmpx * tmpx / T::from_u64(n),\n            sx3: m3::combine(\n                self.n64(),\n                other.n64(),\n                self.sx,\n                other.sx,\n                self.sx2,\n                other.sx2,\n                self.sx3,\n                other.sx3,\n            ),\n            sx4: m4::combine(\n                self.n64(),\n                other.n64(),\n                self.sx,\n                other.sx,\n                self.sx2,\n                other.sx2,\n                self.sx3,\n                other.sx3,\n                self.sx4,\n                other.sx4,\n            ),\n            sy: self.sy + other.sy,\n            sy2: self.sy2 + other.sy2 + self.n64() * other.n64() * tmpy * tmpy / T::from_u64(n),\n            sy3: m3::combine(\n                self.n64(),\n                other.n64(),\n                self.sy,\n                other.sy,\n                self.sy2,\n                other.sy2,\n                self.sy3,\n                other.sy3,\n            ),\n            sy4: m4::combine(\n                self.n64(),\n                other.n64(),\n                self.sy,\n                other.sy,\n                self.sy2,\n                other.sy2,\n                self.sy3,\n                other.sy3,\n                self.sy4,\n                other.sy4,\n            ),\n            sxy: self.sxy + other.sxy + self.n64() * other.n64() * tmpx * tmpy / T::from_u64(n),\n        };\n        if r.has_infinite() && !self.has_infinite() && !other.has_infinite() {\n            return Err(StatsError::DoubleOverflow);\n        }\n        Ok(r)\n    }\n\n    // This is the inverse combine function for use in the window function context when we want to reverse the operation of the normal combine function\n    // for re-aggregation over a window, this is what will get called in tumbling window averages for instance.\n    // As with any window function, returning None will cause a re-calculation, so we do that in several cases where either we're dealing with infinites or we have some potential problems with outlying sums\n    // so here, self is the previously combined StatsSummary, and we're removing the input and returning the part that would have been there before.\n    pub fn remove_combined(&self, remove: StatsSummary2D<T>) -> Option<Self> {\n        let combined = &self; // just to lessen confusion with naming\n                              // handle the trivial n = 0 and equal n cases here, and don't worry about divide by zero errors later.\n        if combined.n == remove.n {\n            return Some(StatsSummary2D::new());\n        } else if remove.n == 0 {\n            return Some(*self);\n        } else if combined.n < remove.n {\n            panic!(); //  given that we're always removing things that we've previously added, we shouldn't be able to get a case where we're removing an n that's larger.\n        }\n        // if the sum we're removing is very large compared to the overall value we need to recalculate, see note on the remove function\n        let thresh = <T as From<f64>>::from(INV_FLOATING_ERROR_THRESHOLD);\n        if remove.sx / combined.sx > thresh || remove.sy / combined.sy > thresh {\n            return None;\n        }\n        let mut part = StatsSummary2D {\n            n: combined.n - remove.n,\n            sx: combined.sx - remove.sx,\n            sy: combined.sy - remove.sy,\n            sx2: T::zero(), //just initialize these, for now.\n            sx3: T::zero(),\n            sx4: T::zero(),\n            sy2: T::zero(),\n            sy3: T::zero(),\n            sy4: T::zero(),\n            sxy: T::zero(),\n        };\n        let tmpx = part.sx / part.n64() - remove.sx / remove.n64(); //gets squared so order doesn't matter\n        let tmpy = part.sy / part.n64() - remove.sy / remove.n64();\n        part.sx2 =\n            combined.sx2 - remove.sx2 - part.n64() * remove.n64() * tmpx * tmpx / combined.n64();\n        part.sx3 = m3::remove_combined(\n            part.n64(),\n            remove.n64(),\n            part.sx,\n            remove.sx,\n            part.sx2,\n            remove.sx2,\n            self.sx3,\n            remove.sx3,\n        );\n        part.sx4 = m4::remove_combined(\n            part.n64(),\n            remove.n64(),\n            part.sx,\n            remove.sx,\n            part.sx2,\n            remove.sx2,\n            part.sx3,\n            remove.sx3,\n            self.sx4,\n            remove.sx4,\n        );\n        part.sy2 =\n            combined.sy2 - remove.sy2 - part.n64() * remove.n64() * tmpy * tmpy / combined.n64();\n        part.sy3 = m3::remove_combined(\n            part.n64(),\n            remove.n64(),\n            part.sy,\n            remove.sy,\n            part.sy2,\n            remove.sy2,\n            self.sy3,\n            remove.sy3,\n        );\n        part.sy4 = m4::remove_combined(\n            part.n64(),\n            remove.n64(),\n            part.sy,\n            remove.sy,\n            part.sy2,\n            remove.sy2,\n            part.sy3,\n            remove.sy3,\n            self.sy4,\n            remove.sy4,\n        );\n        part.sxy =\n            combined.sxy - remove.sxy - part.n64() * remove.n64() * tmpx * tmpy / combined.n64();\n        Some(part)\n    }\n\n    /// offsets all values accumulated in a StatsSummary2D by a given amount in X & Y. This\n    /// only works if all values are offset by that amount. This is used for allowing\n    /// relative calculations in a local region and then allowing them to be combined with\n    /// other regions where all points are offset by the same amount. The main use case\n    /// for now is in the counter case where, when partials are combined you can get a new\n    /// offset for all points in the counter.\n    // Note that when offsetting, the offset of the previous partial  be multiplied by N and added to the Sy value. All the other values are\n    // unaffected because they rely on the expression (Y-Sy/N), (and analogous for the X values) which is basically each value subtracted from the\n    // average of all values and if all values are shifted by a constant, then the average shifts by the same constant so it cancels out:\n    // i.e. If a constant C is added to each Y, then (Y-Sy/N) reduces back to itself as follows:\n    //(Y + C) - (Sy + NC)/N\n    // Y + C - Sy/N - NC/N\n    // Y + C - Sy/N - C\n    // Y - Sy/N\n    pub fn offset(&mut self, offset: XYPair<T>) -> Result<(), StatsError> {\n        self.sx += self.n64() * offset.x;\n        self.sy += self.n64() * offset.y;\n        if self.has_infinite() && offset.x.is_finite() && offset.y.is_finite() {\n            return Err(StatsError::DoubleOverflow);\n        }\n        Ok(())\n    }\n\n    //TODO: Add tests for offsets\n\n    ///returns the sum of squares of both the independent (x) and dependent (y) variables\n    ///as an XYPair, where the sum of squares is defined as: sum(x^2) - sum(x)^2 / n)\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let ssx = (1.0_f64.powi(2) + 2.0_f64.powi(2) + 3.0_f64.powi(2)) - (1.0+2.0+3.0_f64).powi(2)/3.0;\n    /// let ssy = (2.0_f64.powi(2) + 4.0_f64.powi(2) + 6.0_f64.powi(2)) - (2.0+4.0+6.0_f64).powi(2)/3.0;\n    /// let ssp = p.sum_squares().unwrap();\n    /// assert_eq!(ssp.x, ssx);\n    /// assert_eq!(ssp.y, ssy);\n    /// //empty StatsSummary2Ds return None\n    /// assert!(StatsSummary2D::<f64>::new().sum_squares().is_none());\n    /// ```\n    pub fn sum_squares(&self) -> Option<XYPair<T>> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(XYPair {\n            x: self.sx2,\n            y: self.sy2,\n        })\n    }\n    ///returns the \"sum of products\" of the dependent * independent variables sum(x * y) - sum(x) * sum(y) / n\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let s = (2.0 * 1.0 + 4.0 * 2.0 + 6.0 * 3.0) - (2.0 + 4.0 + 6.0)*(1.0 + 2.0 + 3.0)/3.0;\n    /// assert_eq!(p.sumxy().unwrap(), s);\n    /// //empty StatsSummary2Ds return None\n    /// assert!(StatsSummary2D::<f64>::new().sumxy().is_none());\n    /// ```\n    pub fn sumxy(&self) -> Option<T> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(self.sxy)\n    }\n    ///returns the averages of the x and y variables\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let avgx = (1.0 + 2.0 + 3.0)/3.0;\n    /// let avgy = (2.0 + 4.0 + 6.0)/3.0;\n    /// let avgp = p.avg().unwrap();\n    /// assert_eq!(avgp.x, avgx);\n    /// assert_eq!(avgp.y, avgy);\n    /// //empty StatsSummary2Ds return None\n    /// assert!(StatsSummary2D::<f64>::new().avg().is_none());\n    /// ```\n    pub fn avg(&self) -> Option<XYPair<T>> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(XYPair {\n            x: self.sx / self.n64(),\n            y: self.sy / self.n64(),\n        })\n    }\n\n    ///returns the count of inputs as an i64\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    ///\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let s = 3;\n    /// assert_eq!(p.count(), s);\n    /// //empty StatsSummary2Ds return 0 count\n    /// assert_eq!(StatsSummary2D::<f64>::new().count(), 0);\n    /// ```\n    pub fn count(&self) -> i64 {\n        self.n as i64\n    }\n    ///returns the sums of x and y as an XYPair\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let sumx = (1.0 + 2.0 + 3.0);\n    /// let sumy = (2.0 + 4.0 + 6.0);\n    /// let sump = p.sum().unwrap();\n    /// assert_eq!(sump.x, sumx);\n    /// assert_eq!(sump.y, sumy);\n    /// //empty StatsSummary2Ds return None\n    /// assert!(StatsSummary2D::<f64>::new().sum().is_none());\n    /// ```\n    pub fn sum(&self) -> Option<XYPair<T>> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(XYPair {\n            x: self.sx,\n            y: self.sy,\n        })\n    }\n\n    pub fn var_pop(&self) -> Option<XYPair<T>> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(XYPair {\n            x: self.sx2 / self.n64(),\n            y: self.sy2 / self.n64(),\n        })\n    }\n\n    pub fn var_samp(&self) -> Option<XYPair<T>> {\n        if self.n <= 1 {\n            return None;\n        }\n        Some(XYPair {\n            x: self.sx2 / (self.n64() - T::one()),\n            y: self.sy2 / (self.n64() - T::one()),\n        })\n    }\n\n    ///returns the population standard deviation of both the independent and dependent variables as an XYPair\n    pub fn stddev_pop(&self) -> Option<XYPair<T>> {\n        let var = self.var_pop()?;\n        Some(XYPair {\n            x: var.x.sqrt(),\n            y: var.y.sqrt(),\n        })\n    }\n\n    ///returns the sample standard deviation of both the independent and dependent variables as an XYPair\n    pub fn stddev_samp(&self) -> Option<XYPair<T>> {\n        let var = self.var_samp()?;\n        Some(XYPair {\n            x: var.x.sqrt(),\n            y: var.y.sqrt(),\n        })\n    }\n\n    pub fn skewness_pop(&self) -> Option<XYPair<T>> {\n        let stddev = self.stddev_pop()?;\n        Some(XYPair {\n            x: self.sx3 / self.n64() / stddev.x.powi(3),\n            y: self.sy3 / self.n64() / stddev.y.powi(3),\n        })\n    }\n\n    pub fn skewness_samp(&self) -> Option<XYPair<T>> {\n        let stddev = self.stddev_samp()?;\n        Some(XYPair {\n            x: self.sx3 / (self.n64() - T::one()) / stddev.x.powi(3),\n            y: self.sy3 / (self.n64() - T::one()) / stddev.y.powi(3),\n        })\n    }\n\n    pub fn kurtosis_pop(&self) -> Option<XYPair<T>> {\n        let stddev = self.stddev_pop()?;\n        Some(XYPair {\n            x: self.sx4 / self.n64() / stddev.x.powi(4),\n            y: self.sy4 / self.n64() / stddev.y.powi(4),\n        })\n    }\n\n    pub fn kurtosis_samp(&self) -> Option<XYPair<T>> {\n        let stddev = self.stddev_samp()?;\n        Some(XYPair {\n            x: self.sx4 / (self.n64() - T::one()) / stddev.x.powi(4),\n            y: self.sy4 / (self.n64() - T::one()) / stddev.y.powi(4),\n        })\n    }\n\n    /// returns the correlation coefficient, which is the covariance / (stddev(x) * stddev(y))\n    /// Note that it makes no difference whether we choose the sample or\n    /// population covariance and stddev, because we end up with a canceling n or n-1 term. This\n    /// also allows us to reduce our calculation to the sumxy / sqrt(sum_squares(x)*sum_squares(y))\n    pub fn corr(&self) -> Option<T> {\n        // empty StatsSummary2Ds, horizontal or vertical lines should return None\n        if self.n == 0 || self.sx2 == T::zero() || self.sy2 == T::zero() {\n            return None;\n        }\n        Some(self.sxy / (self.sx2 * self.sy2).sqrt())\n    }\n\n    /// returns the slope of the least squares fit line\n    pub fn slope(&self) -> Option<T> {\n        // the case of a single point will usually be triggered by the the second branch of this (which is also a test for a vertical line)\n        //however, in cases where we had an infinite input, we will end up with NaN which is the expected behavior.\n        if self.n == 0 || self.sx2 == T::zero() {\n            return None;\n        }\n        Some(self.sxy / self.sx2)\n    }\n\n    /// returns the intercept of the least squares fit line\n    pub fn intercept(&self) -> Option<T> {\n        if self.n == 0 || self.sx2 == T::zero() {\n            return None;\n        }\n        Some((self.sy - self.sx * self.sxy / self.sx2) / self.n64())\n    }\n\n    /// returns the x intercept of the least squares fit line\n    // y = mx + b (y = 0)\n    // -b = mx\n    // x = -b / m\n    pub fn x_intercept(&self) -> Option<T> {\n        // vertical line does have an x intercept\n        if self.n > 1 && self.sx2 == T::zero() {\n            return Some(self.sx / self.n64());\n        }\n        // horizontal lines have no x intercepts\n        if self.sy2 == T::zero() {\n            return None;\n        }\n        Some(-self.intercept()? / self.slope()?)\n    }\n\n    /// returns the square of the correlation coefficient (aka the coefficient of determination)\n    pub fn determination_coeff(&self) -> Option<T> {\n        if self.n == 0 || self.sx2 == T::zero() {\n            return None;\n        }\n        //horizontal lines return 1.0 error\n        if self.sy2 == T::zero() {\n            return Some(T::one());\n        }\n        Some(self.sxy * self.sxy / (self.sx2 * self.sy2))\n    }\n\n    ///returns the sample covariance: (sumxy()/n-1)\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let s = (2.0 * 1.0 + 4.0 * 2.0 + 6.0 * 3.0) - (2.0 + 4.0 + 6.0)*(1.0 + 2.0 + 3.0)/3.0;\n    /// let s = s/2.0;\n    /// assert_eq!(p.covar_samp().unwrap(), s);\n    /// //empty StatsSummary2Ds return None\n    /// assert!(StatsSummary2D::<f64>::new().covar_samp().is_none());\n    /// ```\n    pub fn covar_samp(&self) -> Option<T> {\n        if self.n <= 1 {\n            return None;\n        }\n        Some(self.sxy / (self.n64() - T::one()))\n    }\n\n    ///returns the population covariance: (sumxy()/n)\n    ///```\n    /// use stats_agg::stats2d::StatsSummary2D;\n    /// use stats_agg::XYPair;\n    /// let p = StatsSummary2D::new_from_vec(vec![XYPair{y:2.0, x:1.0,}, XYPair{y:4.0, x:2.0,}, XYPair{y:6.0, x:3.0,}]).unwrap();\n    /// let s = (2.0 * 1.0 + 4.0 * 2.0 + 6.0 * 3.0) - (2.0 + 4.0 + 6.0)*(1.0 + 2.0 + 3.0)/3.0;\n    /// let s = s/3.0;\n    /// assert_eq!(p.covar_pop().unwrap(), s);\n    /// //empty StatsSummary2Ds return None\n    /// assert!(StatsSummary2D::<f64>::new().covar_pop().is_none());\n    /// ```\n    pub fn covar_pop(&self) -> Option<T> {\n        if self.n == 0 {\n            return None;\n        }\n        Some(self.sxy / self.n64())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    fn tf(f: f64) -> TwoFloat {\n        TwoFloat::new_add(f, 0.0)\n    }\n\n    #[test]\n    fn test_linear() {\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair { y: 2.0, x: 1.0 },\n            XYPair { y: 4.0, x: 2.0 },\n            XYPair { y: 6.0, x: 3.0 },\n        ])\n        .unwrap();\n        assert_eq!(p.slope().unwrap(), 2.0);\n        assert_eq!(p.intercept().unwrap(), 0.0);\n        assert_eq!(p.x_intercept().unwrap(), 0.0);\n\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair { y: 2.0, x: 2.0 },\n            XYPair { y: 4.0, x: 3.0 },\n            XYPair { y: 6.0, x: 4.0 },\n        ])\n        .unwrap();\n        assert_eq!(p.slope().unwrap(), 2.0);\n        assert_eq!(p.intercept().unwrap(), -2.0);\n        assert_eq!(p.x_intercept().unwrap(), 1.0);\n\n        // empty\n        let p: StatsSummary2D<f64> = StatsSummary2D::new();\n        assert_eq!(p.slope(), None);\n        assert_eq!(p.intercept(), None);\n        assert_eq!(p.x_intercept(), None);\n        // singleton\n        let p = StatsSummary2D::new_from_vec(vec![XYPair { y: 2.0, x: 2.0 }]).unwrap();\n        assert_eq!(p.slope(), None);\n        assert_eq!(p.intercept(), None);\n        assert_eq!(p.x_intercept(), None);\n        //vertical\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair { y: 2.0, x: 2.0 },\n            XYPair { y: 4.0, x: 2.0 },\n        ])\n        .unwrap();\n        assert_eq!(p.slope(), None);\n        assert_eq!(p.intercept(), None);\n        assert_eq!(p.x_intercept().unwrap(), 2.0);\n        //horizontal\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair { y: 2.0, x: 2.0 },\n            XYPair { y: 2.0, x: 4.0 },\n        ])\n        .unwrap();\n        assert_eq!(p.slope().unwrap(), 0.0);\n        assert_eq!(p.intercept().unwrap(), 2.0);\n        assert_eq!(p.x_intercept(), None);\n    }\n\n    #[test]\n    fn test_linear_tf() {\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair {\n                y: tf(2.0),\n                x: tf(1.0),\n            },\n            XYPair {\n                y: tf(4.0),\n                x: tf(2.0),\n            },\n            XYPair {\n                y: tf(6.0),\n                x: tf(3.0),\n            },\n        ])\n        .unwrap();\n        assert_eq!(p.slope().unwrap(), tf(2.0));\n        assert_eq!(p.intercept().unwrap(), tf(0.0));\n        assert_eq!(p.x_intercept().unwrap(), tf(0.0));\n\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair {\n                y: tf(2.0),\n                x: tf(2.0),\n            },\n            XYPair {\n                y: tf(4.0),\n                x: tf(3.0),\n            },\n            XYPair {\n                y: tf(6.0),\n                x: tf(4.0),\n            },\n        ])\n        .unwrap();\n        assert_eq!(p.slope().unwrap(), tf(2.0));\n        assert_eq!(p.intercept().unwrap().hi(), -2.0);\n        assert!(p.intercept().unwrap().lo().abs() < f64::EPSILON);\n        assert_eq!(p.x_intercept().unwrap().hi(), 1.0);\n        assert!(p.x_intercept().unwrap().lo().abs() < f64::EPSILON);\n\n        // empty\n        let p: StatsSummary2D<TwoFloat> = StatsSummary2D::new();\n        assert_eq!(p.slope(), None);\n        assert_eq!(p.intercept(), None);\n        assert_eq!(p.x_intercept(), None);\n        // singleton\n        let p = StatsSummary2D::new_from_vec(vec![XYPair {\n            y: tf(2.0),\n            x: tf(2.0),\n        }])\n        .unwrap();\n        assert_eq!(p.slope(), None);\n        assert_eq!(p.intercept(), None);\n        assert_eq!(p.x_intercept(), None);\n        //vertical\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair {\n                y: tf(2.0),\n                x: tf(2.0),\n            },\n            XYPair {\n                y: tf(4.0),\n                x: tf(2.0),\n            },\n        ])\n        .unwrap();\n        assert_eq!(p.slope(), None);\n        assert_eq!(p.intercept(), None);\n        assert_eq!(p.x_intercept().unwrap(), tf(2.0));\n        //horizontal\n        let p = StatsSummary2D::new_from_vec(vec![\n            XYPair {\n                y: tf(2.0),\n                x: tf(2.0),\n            },\n            XYPair {\n                y: tf(2.0),\n                x: tf(4.0),\n            },\n        ])\n        .unwrap();\n        assert_eq!(p.slope().unwrap(), tf(0.0));\n        assert_eq!(p.intercept().unwrap(), tf(2.0));\n        assert_eq!(p.x_intercept(), None);\n    }\n}\n"
  },
  {
    "path": "crates/t-digest/Cargo.toml",
    "content": "[package]\nname = \"tdigest\"\nversion = \"0.2.2\"\nedition = \"2021\"\n# based on: https://github.com/MnO2/t-digest\"\n\n[dependencies]\nflat_serialize = {path=\"../flat_serialize/flat_serialize\"}\nflat_serialize_macro = {path=\"../flat_serialize/flat_serialize_macro\"}\nordered-float = {version = \"1.0\", features = [\"serde\"] }\nron = \"0.6.0\"\nserde = { version = \"1.0\", features = [\"derive\"] }\n\n[dev-dependencies]\nquickcheck = \"1\"\nquickcheck_macros = \"1\"\n"
  },
  {
    "path": "crates/t-digest/src/lib.rs",
    "content": "// Based on https://github.com/MnO2/t-digest/blob/master/src/lib.rs\n// as of commit 66d7c19d32c1547daa628f1d9f12178a686ba022\n\n//! T-Digest algorithm in rust\n//!\n//! ## Installation\n//!\n//! Add this to your `Cargo.toml`:\n//!\n//! ```toml\n//! [dependencies]\n//! tdigest = \"0.2\"\n//! ```\n//!\n//! then you are good to go. If you are using Rust 2015 you have to ``extern crate tdigest`` to your crate root as well.\n//!\n//! ## Example\n//!\n//! ```rust\n//! use tdigest::TDigest;\n//!\n//! let t = TDigest::new_with_size(100);\n//! let values: Vec<f64> = (1..=1_000_000).map(f64::from).collect();\n//!\n//! let t = t.merge_sorted(values);\n//!\n//! let ans = t.estimate_quantile(0.99);\n//! let expected: f64 = 990_000.0;\n//!\n//! let percentage: f64 = (expected - ans).abs() / expected;\n//! assert!(percentage < 0.01);\n//! ```\n\nuse ordered_float::OrderedFloat;\nuse std::cmp::Ordering;\n#[cfg(test)]\nuse std::collections::HashSet;\n\nuse flat_serialize_macro::FlatSerializable;\n\nuse serde::{Deserialize, Serialize};\n\n#[cfg(test)]\nextern crate quickcheck;\n#[cfg(test)]\n#[macro_use(quickcheck)]\nextern crate quickcheck_macros;\n\n/// Centroid implementation to the cluster mentioned in the paper.\n#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize, FlatSerializable)]\n#[repr(C)]\npub struct Centroid {\n    mean: OrderedFloat<f64>,\n    weight: u64,\n}\n\nimpl PartialOrd for Centroid {\n    fn partial_cmp(&self, other: &Centroid) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Ord for Centroid {\n    fn cmp(&self, other: &Centroid) -> Ordering {\n        self.mean.cmp(&other.mean)\n    }\n}\n\nimpl Centroid {\n    pub fn new(mean: f64, weight: u64) -> Self {\n        Centroid {\n            mean: OrderedFloat::from(mean),\n            weight,\n        }\n    }\n\n    #[inline]\n    pub fn mean(&self) -> f64 {\n        self.mean.into_inner()\n    }\n\n    #[inline]\n    pub fn weight(&self) -> u64 {\n        self.weight\n    }\n\n    pub fn add(&mut self, sum: f64, weight: u64) -> f64 {\n        let weight_: u64 = self.weight;\n        let mean_: f64 = self.mean.into_inner();\n\n        let new_sum: f64 = sum + weight_ as f64 * mean_;\n        let new_weight: u64 = weight_ + weight;\n        self.weight = new_weight;\n        self.mean = OrderedFloat::from(new_sum / new_weight as f64);\n        new_sum\n    }\n}\n\nimpl Default for Centroid {\n    fn default() -> Self {\n        Centroid {\n            mean: OrderedFloat::from(0.0),\n            weight: 1,\n        }\n    }\n}\n\n/// T-Digest to be operated on.\n#[derive(Debug, PartialEq, Eq, Clone, Serialize, Deserialize)]\npub struct TDigest {\n    centroids: Vec<Centroid>,\n    max_size: usize,\n    sum: OrderedFloat<f64>,\n    count: u64,\n    max: OrderedFloat<f64>,\n    min: OrderedFloat<f64>,\n}\n\nimpl TDigest {\n    pub fn new_with_size(max_size: usize) -> Self {\n        TDigest {\n            centroids: Vec::new(),\n            max_size,\n            sum: OrderedFloat::from(0.0),\n            count: 0,\n            max: OrderedFloat::from(f64::NAN),\n            min: OrderedFloat::from(f64::NAN),\n        }\n    }\n\n    pub fn new(\n        centroids: Vec<Centroid>,\n        sum: f64,\n        count: u64,\n        max: f64,\n        min: f64,\n        max_size: usize,\n    ) -> Self {\n        if centroids.len() <= max_size {\n            TDigest {\n                centroids,\n                max_size,\n                sum: OrderedFloat::from(sum),\n                count,\n                max: OrderedFloat::from(max),\n                min: OrderedFloat::from(min),\n            }\n        } else {\n            let sz = centroids.len();\n            let digests: Vec<TDigest> = vec![\n                TDigest::new_with_size(max_size),\n                TDigest::new(centroids, sum, count, max, min, sz),\n            ];\n\n            Self::merge_digests(digests)\n        }\n    }\n\n    pub fn raw_centroids(&self) -> &[Centroid] {\n        &self.centroids\n    }\n\n    #[inline]\n    pub fn mean(&self) -> f64 {\n        let sum_: f64 = self.sum.into_inner();\n\n        if self.count > 0 {\n            sum_ / self.count as f64\n        } else {\n            0.0\n        }\n    }\n\n    #[inline]\n    pub fn sum(&self) -> f64 {\n        self.sum.into_inner()\n    }\n\n    #[inline]\n    pub fn count(&self) -> u64 {\n        self.count\n    }\n\n    #[inline]\n    pub fn max(&self) -> f64 {\n        self.max.into_inner()\n    }\n\n    #[inline]\n    pub fn min(&self) -> f64 {\n        self.min.into_inner()\n    }\n\n    #[inline]\n    pub fn is_empty(&self) -> bool {\n        self.centroids.is_empty()\n    }\n\n    #[inline]\n    pub fn max_size(&self) -> usize {\n        self.max_size\n    }\n\n    #[inline]\n    pub fn num_buckets(&self) -> usize {\n        self.centroids.len()\n    }\n\n    pub fn format_for_postgres(&self) -> String {\n        /// Mimics the version-1 serialization format the extension uses.  TODO don't!\n        #[derive(Serialize)]\n        struct Hack {\n            version: u32,\n            buckets: usize,\n            max_buckets: usize,\n            count: u64,\n            sum: f64,\n            min: f64,\n            max: f64,\n            centroids: Vec<Centroid>,\n        }\n\n        let max_buckets = self.max_size();\n        let centroids = self.raw_centroids();\n        ron::to_string(&Hack {\n            version: 1,\n            max_buckets,\n            buckets: centroids.len(),\n            count: self.count(),\n            sum: self.sum(),\n            min: self.min(),\n            max: self.max(),\n            centroids: centroids.to_vec(),\n        })\n        .unwrap()\n    }\n}\n\nimpl Default for TDigest {\n    fn default() -> Self {\n        TDigest {\n            centroids: Vec::new(),\n            max_size: 100,\n            sum: OrderedFloat::from(0.0),\n            count: 0,\n            max: OrderedFloat::from(f64::NAN),\n            min: OrderedFloat::from(f64::NAN),\n        }\n    }\n}\n\nimpl TDigest {\n    fn k_to_q(k: f64, d: f64) -> f64 {\n        let k_div_d = k / d;\n        if k_div_d >= 0.5 {\n            let base = 1.0 - k_div_d;\n            1.0 - 2.0 * base * base\n        } else {\n            2.0 * k_div_d * k_div_d\n        }\n    }\n\n    pub fn merge_unsorted(&self, unsorted_values: Vec<f64>) -> TDigest {\n        let mut sorted_values: Vec<OrderedFloat<f64>> = unsorted_values\n            .into_iter()\n            .map(OrderedFloat::from)\n            .collect();\n        sorted_values.sort();\n        let sorted_values = sorted_values.into_iter().map(|f| f.into_inner()).collect();\n\n        self.merge_sorted(sorted_values)\n    }\n\n    // Allow f64 overflow to create centroids with infinite mean, but make sure our min/max are updated\n    fn update_bounds_on_overflow(\n        value: OrderedFloat<f64>,\n        lower_bound: &mut OrderedFloat<f64>,\n        upper_bound: &mut OrderedFloat<f64>,\n    ) {\n        if value < *lower_bound {\n            *lower_bound = value;\n        }\n        if value > *upper_bound {\n            *upper_bound = value;\n        }\n    }\n\n    pub fn merge_sorted(&self, sorted_values: Vec<f64>) -> TDigest {\n        if sorted_values.is_empty() {\n            return self.clone();\n        }\n\n        let mut result = TDigest::new_with_size(self.max_size());\n        result.count = self.count() + (sorted_values.len() as u64);\n\n        let maybe_min = OrderedFloat::from(*sorted_values.first().unwrap());\n        let maybe_max = OrderedFloat::from(*sorted_values.last().unwrap());\n\n        if self.count() > 0 {\n            result.min = std::cmp::min(self.min, maybe_min);\n            result.max = std::cmp::max(self.max, maybe_max);\n        } else {\n            result.min = maybe_min;\n            result.max = maybe_max;\n        }\n\n        let mut compressed: Vec<Centroid> = Vec::with_capacity(self.max_size);\n\n        let mut k_limit: f64 = 1.0;\n        let mut q_limit_times_count: f64 =\n            Self::k_to_q(k_limit, self.max_size as f64) * result.count as f64;\n        k_limit += 1.0;\n\n        let mut iter_centroids = self.centroids.iter().peekable();\n        let mut iter_sorted_values = sorted_values.iter().peekable();\n\n        let mut curr: Centroid = if let Some(c) = iter_centroids.peek() {\n            let curr = **iter_sorted_values.peek().unwrap();\n            if c.mean() < curr {\n                iter_centroids.next().unwrap().clone()\n            } else {\n                Centroid::new(*iter_sorted_values.next().unwrap(), 1)\n            }\n        } else {\n            Centroid::new(*iter_sorted_values.next().unwrap(), 1)\n        };\n\n        let mut weight_so_far: u64 = curr.weight();\n\n        let mut sums_to_merge: f64 = 0.0;\n        let mut weights_to_merge: u64 = 0;\n\n        while iter_centroids.peek().is_some() || iter_sorted_values.peek().is_some() {\n            let next: Centroid = if let Some(c) = iter_centroids.peek() {\n                if iter_sorted_values.peek().is_none()\n                    || c.mean() < **iter_sorted_values.peek().unwrap()\n                {\n                    iter_centroids.next().unwrap().clone()\n                } else {\n                    Centroid::new(*iter_sorted_values.next().unwrap(), 1)\n                }\n            } else {\n                Centroid::new(*iter_sorted_values.next().unwrap(), 1)\n            };\n\n            let next_sum: f64 = next.mean() * next.weight() as f64;\n            weight_so_far += next.weight();\n\n            if weight_so_far as f64 <= q_limit_times_count {\n                sums_to_merge += next_sum;\n                weights_to_merge += next.weight();\n            } else {\n                result.sum = OrderedFloat::from(\n                    result.sum.into_inner() + curr.add(sums_to_merge, weights_to_merge),\n                );\n                sums_to_merge = 0.0;\n                weights_to_merge = 0;\n                TDigest::update_bounds_on_overflow(curr.mean, &mut result.min, &mut result.max);\n\n                compressed.push(curr.clone());\n                q_limit_times_count =\n                    Self::k_to_q(k_limit, self.max_size as f64) * result.count() as f64;\n                k_limit += 1.0;\n                curr = next;\n            }\n        }\n\n        result.sum =\n            OrderedFloat::from(result.sum.into_inner() + curr.add(sums_to_merge, weights_to_merge));\n        TDigest::update_bounds_on_overflow(curr.mean, &mut result.min, &mut result.max);\n        compressed.push(curr);\n        compressed.shrink_to_fit();\n        compressed.sort();\n\n        result.centroids = compressed;\n        result\n    }\n\n    fn external_merge(centroids: &mut [Centroid], first: usize, middle: usize, last: usize) {\n        let mut result: Vec<Centroid> = Vec::with_capacity(centroids.len());\n\n        let mut i = first;\n        let mut j = middle;\n\n        while i < middle && j < last {\n            match centroids[i].cmp(&centroids[j]) {\n                Ordering::Less => {\n                    result.push(centroids[i].clone());\n                    i += 1;\n                }\n                Ordering::Greater => {\n                    result.push(centroids[j].clone());\n                    j += 1;\n                }\n                Ordering::Equal => {\n                    result.push(centroids[i].clone());\n                    i += 1;\n                }\n            }\n        }\n\n        while i < middle {\n            result.push(centroids[i].clone());\n            i += 1;\n        }\n\n        while j < last {\n            result.push(centroids[j].clone());\n            j += 1;\n        }\n\n        i = first;\n        for centroid in result.into_iter() {\n            centroids[i] = centroid;\n            i += 1;\n        }\n    }\n\n    // Merge multiple T-Digests\n    pub fn merge_digests(digests: Vec<TDigest>) -> TDigest {\n        let n_centroids: usize = digests.iter().map(|d| d.centroids.len()).sum();\n        if n_centroids == 0 {\n            return TDigest::default();\n        }\n\n        // TODO should this be the smaller of the sizes?\n        let max_size = digests.first().unwrap().max_size;\n        let mut centroids: Vec<Centroid> = Vec::with_capacity(n_centroids);\n        let mut starts: Vec<usize> = Vec::with_capacity(digests.len());\n\n        let mut count: u64 = 0;\n        let mut min = OrderedFloat::from(f64::INFINITY);\n        let mut max = OrderedFloat::from(f64::NEG_INFINITY);\n\n        let mut start: usize = 0;\n        for digest in digests.into_iter() {\n            starts.push(start);\n\n            let curr_count: u64 = digest.count();\n            if curr_count > 0 {\n                min = std::cmp::min(min, digest.min);\n                max = std::cmp::max(max, digest.max);\n                count += curr_count;\n                for centroid in digest.centroids {\n                    centroids.push(centroid);\n                    start += 1;\n                }\n            }\n        }\n\n        let mut digests_per_block: usize = 1;\n        while digests_per_block < starts.len() {\n            for i in (0..starts.len()).step_by(digests_per_block * 2) {\n                if i + digests_per_block < starts.len() {\n                    let first = starts[i];\n                    let middle = starts[i + digests_per_block];\n                    let last = if i + 2 * digests_per_block < starts.len() {\n                        starts[i + 2 * digests_per_block]\n                    } else {\n                        centroids.len()\n                    };\n\n                    debug_assert!(first <= middle && middle <= last);\n                    Self::external_merge(&mut centroids, first, middle, last);\n                }\n            }\n\n            digests_per_block *= 2;\n        }\n\n        let mut result = TDigest::new_with_size(max_size);\n        let mut compressed: Vec<Centroid> = Vec::with_capacity(max_size);\n\n        let mut k_limit: f64 = 1.0;\n        let mut q_limit_times_count: f64 = Self::k_to_q(k_limit, max_size as f64) * (count as f64);\n\n        let mut iter_centroids = centroids.iter_mut();\n        let mut curr = iter_centroids.next().unwrap();\n        let mut weight_so_far: u64 = curr.weight();\n        let mut sums_to_merge: f64 = 0.0;\n        let mut weights_to_merge: u64 = 0;\n\n        for centroid in iter_centroids {\n            weight_so_far += centroid.weight();\n\n            if weight_so_far as f64 <= q_limit_times_count {\n                sums_to_merge += centroid.mean() * centroid.weight() as f64;\n                weights_to_merge += centroid.weight();\n            } else {\n                result.sum = OrderedFloat::from(\n                    result.sum.into_inner() + curr.add(sums_to_merge, weights_to_merge),\n                );\n                sums_to_merge = 0.0;\n                weights_to_merge = 0;\n                TDigest::update_bounds_on_overflow(curr.mean, &mut min, &mut max);\n                compressed.push(curr.clone());\n                q_limit_times_count = Self::k_to_q(k_limit, max_size as f64) * (count as f64);\n                k_limit += 1.0;\n                curr = centroid;\n            }\n        }\n\n        result.sum =\n            OrderedFloat::from(result.sum.into_inner() + curr.add(sums_to_merge, weights_to_merge));\n        TDigest::update_bounds_on_overflow(curr.mean, &mut min, &mut max);\n        compressed.push(curr.clone());\n        compressed.shrink_to_fit();\n        compressed.sort();\n\n        result.count = count;\n        result.min = min;\n        result.max = max;\n        result.centroids = compressed;\n        result\n    }\n\n    /// Given a value estimate the corresponding quantile in a digest\n    pub fn estimate_quantile_at_value(&self, v: f64) -> f64 {\n        if self.centroids.is_empty() {\n            return 0.0;\n        }\n\n        if v < self.min.into_inner() {\n            return 0.0;\n        }\n\n        if v > self.max.into_inner() {\n            return 1.0;\n        }\n\n        let mut low_bound = self.min.into_inner();\n        let mut low_weight = 0;\n        let mut hi_bound = self.max.into_inner();\n        let mut hi_weight = 0;\n        let mut accum_weight = 0;\n\n        for cent in &self.centroids {\n            if v < cent.mean.into_inner() {\n                hi_bound = cent.mean.into_inner();\n                hi_weight = cent.weight;\n                break;\n            }\n            low_bound = cent.mean.into_inner();\n            low_weight = cent.weight;\n            accum_weight += low_weight;\n        }\n\n        let weighted_midpoint = low_bound\n            + (hi_bound - low_bound) * low_weight as f64 / (low_weight + hi_weight) as f64;\n        if v > weighted_midpoint {\n            (accum_weight as f64\n                + (v - weighted_midpoint) / (hi_bound - weighted_midpoint) * hi_weight as f64 / 2.0)\n                / self.count as f64\n        } else {\n            (accum_weight as f64\n                - (weighted_midpoint - v) / (weighted_midpoint - low_bound) * low_weight as f64\n                    / 2.0)\n                / self.count as f64\n        }\n    }\n\n    /// To estimate the value located at `q` quantile\n    pub fn estimate_quantile(&self, q: f64) -> f64 {\n        if self.centroids.is_empty() {\n            return 0.0;\n        }\n\n        let rank: f64 = q * self.count as f64;\n\n        let mut pos: usize;\n        let mut t: u64;\n        if q > 0.5 {\n            if q >= 1.0 {\n                return self.max();\n            }\n\n            pos = 0;\n            t = self.count;\n\n            for (k, centroid) in self.centroids.iter().enumerate().rev() {\n                t -= centroid.weight();\n\n                if rank >= t as f64 {\n                    pos = k;\n                    break;\n                }\n            }\n        } else {\n            if q <= 0.0 || rank <= 1.0 {\n                return self.min();\n            }\n\n            pos = self.centroids.len() - 1;\n            t = 0;\n\n            for (k, centroid) in self.centroids.iter().enumerate() {\n                if rank < (t + centroid.weight()) as f64 {\n                    pos = k;\n                    break;\n                }\n\n                t += centroid.weight();\n            }\n        }\n\n        // At this point pos indexes the centroid containing the desired rank and t is the combined weight of all buckets < pos\n        // With this we can determine the location of our target rank within the range covered by centroid 'pos'\n        let centroid_weight = (rank - t as f64) / self.centroids[pos].weight() as f64;\n\n        // Now we use that location to interpolate the desired value between the centroid mean and the weighted midpoint between the next centroid in the direction of the target rank.\n        let diff = centroid_weight - 0.5;\n        return if diff.abs() < f64::EPSILON {\n            self.centroids[pos].mean()\n        } else if diff.is_sign_negative() {\n            let weighted_lower_bound = if pos == 0 {\n                weighted_average(\n                    self.min(),\n                    0,\n                    self.centroids[pos].mean(),\n                    self.centroids[pos].weight(),\n                )\n            } else {\n                weighted_average(\n                    self.centroids[pos - 1].mean(),\n                    self.centroids[pos - 1].weight(),\n                    self.centroids[pos].mean(),\n                    self.centroids[pos].weight(),\n                )\n            };\n\n            interpolate(\n                weighted_lower_bound,\n                self.centroids[pos].mean(),\n                centroid_weight * 2.0,\n            )\n        } else {\n            let weighted_upper_bound = if pos == self.centroids.len() - 1 {\n                weighted_average(\n                    self.centroids[pos].mean(),\n                    self.centroids[pos].weight(),\n                    self.max(),\n                    0,\n                )\n            } else {\n                weighted_average(\n                    self.centroids[pos].mean(),\n                    self.centroids[pos].weight(),\n                    self.centroids[pos + 1].mean(),\n                    self.centroids[pos + 1].weight(),\n                )\n            };\n            interpolate(\n                self.centroids[pos].mean(),\n                weighted_upper_bound,\n                (centroid_weight - 0.5) * 2.0,\n            )\n        };\n\n        // Helper functions for quantile calculation\n\n        // Given two points and their relative weights, return the weight midpoint (i.e. if p2 is twice the weight of p1, the midpoint will be twice as close to p2 as to p1)\n        fn weighted_average(p1: f64, p1_weight: u64, p2: f64, p2_weight: u64) -> f64 {\n            interpolate(p1, p2, p1_weight as f64 / (p1_weight + p2_weight) as f64)\n        }\n\n        // Given two points and a weight in the range [0,1], return p1 + weight * (p2-p1)\n        fn interpolate(p1: f64, p2: f64, weight: f64) -> f64 {\n            // We always call this with p2 >= p1 and ensuring this reduces the cases we have to match on\n            debug_assert!(OrderedFloat::from(p2) >= OrderedFloat::from(p1));\n            // Not being able to match on floats makes this match much uglier than it should be\n            match (p1.is_infinite(), p2.is_infinite(), p1.is_sign_positive(), !p2.is_sign_negative()) {\n                (true, true, false, true) /* (f64::NEG_INFINITY, f64::INFINITY) */ => f64::NAN, // This is a stupid case, and the only time we'll see quantile return a NaN\n                (true, _, false, _) /* (f64::NEG_INFINITY, _) */ => f64::NEG_INFINITY,\n                (_, true, _, true) /* (_, f64::INFINITY) */ => f64::INFINITY,\n                _ => p1 + (p2 - p1) * weight\n            }\n        }\n    }\n}\n\n// This is a tdigest object paired\n// with a vector of values that still need to be inserted.\n#[derive(Clone, Debug, Default, Deserialize, Serialize)]\npub struct Builder {\n    #[serde(skip)]\n    buffer: Vec<f64>,\n    digested: TDigest,\n}\n\nimpl From<TDigest> for Builder {\n    fn from(digested: TDigest) -> Self {\n        Self {\n            digested,\n            ..Default::default()\n        }\n    }\n}\n\nimpl Builder {\n    pub fn with_size(size: usize) -> Self {\n        Self::from(TDigest::new_with_size(size))\n    }\n\n    // Add a new value, recalculate the digest if we've crossed a threshold.\n    // TODO threshold is currently set to number of digest buckets, should this be adjusted\n    pub fn push(&mut self, value: f64) {\n        self.buffer.push(value);\n        if self.buffer.len() >= self.digested.max_size() {\n            self.digest()\n        }\n    }\n\n    // Update the digest with all accumulated values.\n    fn digest(&mut self) {\n        if self.buffer.is_empty() {\n            return;\n        }\n        let new = std::mem::take(&mut self.buffer);\n        self.digested = self.digested.merge_unsorted(new)\n    }\n\n    pub fn build(&mut self) -> TDigest {\n        self.digest();\n        std::mem::take(&mut self.digested)\n    }\n\n    pub fn merge(&mut self, other: Self) {\n        assert_eq!(self.digested.max_size(), other.digested.max_size());\n        let digvec = vec![std::mem::take(&mut self.digested), other.digested];\n        if !self.buffer.is_empty() {\n            digvec[0].merge_unsorted(std::mem::take(&mut self.buffer));\n        }\n        if !other.buffer.is_empty() {\n            digvec[1].merge_unsorted(other.buffer);\n        }\n        self.digested = TDigest::merge_digests(digvec);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_centroid_addition_regression() {\n        //https://github.com/MnO2/t-digest/pull/1\n\n        let vals = vec![1.0, 1.0, 1.0, 2.0, 1.0, 1.0];\n        let mut t = TDigest::new_with_size(10);\n\n        for v in vals {\n            t = t.merge_unsorted(vec![v]);\n        }\n\n        let ans = t.estimate_quantile(0.5);\n        let expected: f64 = 1.0;\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.95);\n        let expected: f64 = 2.0;\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n    }\n\n    #[test]\n    fn test_merge_sorted_against_uniform_distro() {\n        let t = TDigest::new_with_size(100);\n        let values: Vec<f64> = (1..=1_000_000).map(f64::from).collect();\n\n        let t = t.merge_sorted(values);\n\n        let ans = t.estimate_quantile(1.0);\n        let expected: f64 = 1_000_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.99);\n        let expected: f64 = 990_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.01);\n        let expected: f64 = 10_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.0);\n        let expected: f64 = 1.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.5);\n        let expected: f64 = 500_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n    }\n\n    #[test]\n    fn test_merge_unsorted_against_uniform_distro() {\n        let t = TDigest::new_with_size(100);\n        let values: Vec<f64> = (1..=1_000_000).map(f64::from).collect();\n\n        let t = t.merge_unsorted(values);\n\n        let ans = t.estimate_quantile(1.0);\n        let expected: f64 = 1_000_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.99);\n        let expected: f64 = 990_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.01);\n        let expected: f64 = 10_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.0);\n        let expected: f64 = 1.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.5);\n        let expected: f64 = 500_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n    }\n\n    #[test]\n    fn test_merge_sorted_against_skewed_distro() {\n        let t = TDigest::new_with_size(100);\n        let mut values: Vec<f64> = (1..=600_000).map(f64::from).collect();\n        values.resize(values.len() + 400_000, 1_000_000.0);\n\n        let t = t.merge_sorted(values);\n\n        let ans = t.estimate_quantile(0.99);\n        let expected: f64 = 1_000_000.0;\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.01);\n        let expected: f64 = 10_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.5);\n        let expected: f64 = 500_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n    }\n\n    #[test]\n    fn test_merge_unsorted_against_skewed_distro() {\n        let t = TDigest::new_with_size(100);\n        let mut values: Vec<f64> = (1..=600_000).map(f64::from).collect();\n        values.resize(values.len() + 400_000, 1_000_000.0);\n\n        let t = t.merge_unsorted(values);\n\n        let ans = t.estimate_quantile(0.99);\n        let expected: f64 = 1_000_000.0;\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.01);\n        let expected: f64 = 10_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.5);\n        let expected: f64 = 500_000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n    }\n\n    #[test]\n    fn test_merge_digests() {\n        let mut digests: Vec<TDigest> = Vec::new();\n\n        for _ in 1..=100 {\n            let t = TDigest::new_with_size(100);\n            let values: Vec<f64> = (1..=1_000).map(f64::from).collect();\n            let t = t.merge_sorted(values);\n            digests.push(t)\n        }\n\n        let t = TDigest::merge_digests(digests);\n\n        let ans = t.estimate_quantile(1.0);\n        let expected: f64 = 1000.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.99);\n        let expected: f64 = 990.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.01);\n        let expected: f64 = 10.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.2);\n\n        let ans = t.estimate_quantile(0.0);\n        let expected: f64 = 1.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n\n        let ans = t.estimate_quantile(0.5);\n        let expected: f64 = 500.0;\n\n        let percentage: f64 = (expected - ans).abs() / expected;\n        assert!(percentage < 0.01);\n    }\n\n    #[test]\n    fn test_quantile_and_value_estimates() {\n        let t = TDigest::new_with_size(100);\n        let values: Vec<f64> = (1..=10000).map(|v| f64::from(v) / 100.0).collect();\n\n        let t = t.merge_sorted(values);\n\n        for i in 1..=100 {\n            let value = i as f64;\n            let quantile = value / 100.0;\n\n            let test_value = t.estimate_quantile(quantile);\n            let test_quant = t.estimate_quantile_at_value(value);\n\n            let percentage = (test_value - value).abs() / value;\n            assert!(\n                percentage < 0.01,\n                \"Exceeded 1% error on quantile {}: expected {}, received {} (error% {})\",\n                quantile,\n                value,\n                test_value,\n                (test_value - value).abs() / value * 100.0\n            );\n            let percentage = (test_quant - quantile).abs() / quantile;\n            assert!(\n                percentage < 0.01,\n                \"Exceeded 1% error on quantile at value {}: expected {}, received {} (error% {})\",\n                value,\n                quantile,\n                test_quant,\n                (test_quant - quantile).abs() / quantile * 100.0\n            );\n\n            let test = t.estimate_quantile_at_value(t.estimate_quantile(quantile));\n            let percentage = (test - quantile).abs() / quantile;\n            assert!(percentage < 0.001);\n        }\n    }\n\n    #[test]\n    fn test_buffered_merge() {\n        let mut digested = TDigest::new_with_size(100);\n        let mut buffer = vec![];\n        for i in 1..=100 {\n            buffer.push(i as f64);\n            if buffer.len() >= digested.max_size() {\n                let new = std::mem::take(&mut buffer);\n                digested = digested.merge_unsorted(new)\n            }\n        }\n        if !buffer.is_empty() {\n            digested = digested.merge_unsorted(buffer)\n        }\n        let estimate = digested.estimate_quantile(0.99);\n        assert_eq!(estimate, 99.5);\n    }\n\n    use quickcheck::*;\n\n    #[derive(Hash, PartialEq, Eq, PartialOrd, Ord, Copy, Clone, Debug)]\n    struct OrderedF64(OrderedFloat<f64>);\n\n    impl Arbitrary for OrderedF64 {\n        fn arbitrary(g: &mut Gen) -> Self {\n            OrderedF64(f64::arbitrary(g).into())\n        }\n    }\n\n    #[quickcheck]\n    fn fuzzing_test(\n        batch1: HashSet<OrderedF64>,\n        batch2: HashSet<OrderedF64>,\n        batch3: HashSet<OrderedF64>,\n        batch4: HashSet<OrderedF64>,\n    ) -> TestResult {\n        let batch1: Vec<f64> = batch1\n            .into_iter()\n            .map(|x| x.0.into())\n            .filter(|x: &f64| !x.is_nan())\n            .collect();\n        let batch2: Vec<f64> = batch2\n            .into_iter()\n            .map(|x| x.0.into())\n            .filter(|x: &f64| !x.is_nan())\n            .collect();\n        let batch3: Vec<f64> = batch3\n            .into_iter()\n            .map(|x| x.0.into())\n            .filter(|x: &f64| !x.is_nan())\n            .collect();\n        let batch4: Vec<f64> = batch4\n            .into_iter()\n            .map(|x| x.0.into())\n            .filter(|x: &f64| !x.is_nan())\n            .collect();\n        let digest1 = TDigest::new_with_size(20).merge_unsorted(batch1.clone());\n        let digest1 = digest1.merge_unsorted(batch2.clone());\n        let digest2 = TDigest::new_with_size(20).merge_unsorted(batch3.clone());\n        let digest2 = digest2.merge_unsorted(batch4.clone());\n\n        let digest = TDigest::merge_digests(vec![digest1, digest2]);\n\n        let quantile_tests = [0.01, 0.1, 0.25, 0.5, 0.6, 0.8, 0.95];\n        let tolerated_percentile_error = [0.010001, 0.100001, 0.2, 0.30, 0.275, 0.1725, 0.050001]; // .000001 cases are to handle rounding errors on cases that might return infinities\n\n        let mut master: Vec<f64> = batch1\n            .iter()\n            .chain(batch2.iter())\n            .chain(batch3.iter())\n            .chain(batch4.iter())\n            .copied()\n            .collect();\n        master.sort_by(|a, b| a.partial_cmp(b).unwrap());\n\n        if master.len() < 100 {\n            return TestResult::discard();\n        }\n\n        for i in 0..quantile_tests.len() {\n            let quantile = quantile_tests[i];\n            let error_bound = tolerated_percentile_error[i];\n\n            let test_val = digest.estimate_quantile(quantile);\n            let target_idx = quantile * master.len() as f64;\n            let target_allowed_error = master.len() as f64 * error_bound;\n            let mut test_idx = 0;\n            if test_val != f64::INFINITY {\n                while test_idx < master.len() && master[test_idx] < test_val {\n                    test_idx += 1;\n                }\n            } else {\n                // inequality checking against infinity is wonky\n                test_idx = master.len();\n            }\n            // test idx is now the idx of the smallest element >= test_val (and yes, this could be done faster with binary search)\n\n            assert!((test_idx as f64) >= target_idx - target_allowed_error && (test_idx as f64) <= target_idx + target_allowed_error, \"testing {} quantile returned {}, there are {} values lower than this, target range {} +/- {}\", quantile, test_val, test_idx, target_idx, target_allowed_error);\n        }\n\n        TestResult::passed()\n    }\n}\n"
  },
  {
    "path": "crates/t-digest-lib/Cargo.toml",
    "content": "[package]\nname = \"tdigest-lib\"\nversion = \"0.0.0\"\nedition = \"2021\"\n\n[lib]\nname = \"timescaledb_toolkit_tdigest\"\ncrate-type = [\"cdylib\", \"staticlib\"]\n\n[dependencies]\nlibc = \"0.2.135\"\n\ntdigest = { path=\"../t-digest\" }\n"
  },
  {
    "path": "crates/t-digest-lib/src/lib.rs",
    "content": "// There is no safety here:  it's all in the hands of the caller, bless their heart.\n#![allow(clippy::missing_safety_doc)]\n\n#[unsafe(no_mangle)]\npub extern \"C\" fn timescaledb_toolkit_tdigest_builder_with_size(\n    size: usize,\n) -> Box<tdigest::Builder> {\n    Box::new(tdigest::Builder::with_size(size))\n}\n\n#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn timescaledb_toolkit_tdigest_push(\n    builder: *mut tdigest::Builder,\n    value: f64,\n) {\n    (*builder).push(value)\n}\n\n// TODO Don't abort the process if `builder` and `other` weren't created with the same size.\n#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn timescaledb_toolkit_tdigest_merge(\n    builder: *mut tdigest::Builder,\n    other: Box<tdigest::Builder>,\n) {\n    let other = *other;\n    (*builder).merge(other)\n}\n\n#[unsafe(no_mangle)]\npub extern \"C\" fn timescaledb_toolkit_tdigest_builder_free(_: Box<tdigest::Builder>) {}\n\n#[unsafe(no_mangle)]\npub extern \"C\" fn timescaledb_toolkit_tdigest_build(\n    mut builder: Box<tdigest::Builder>,\n) -> Box<tdigest::TDigest> {\n    Box::new(builder.build())\n}\n\n#[unsafe(no_mangle)]\npub extern \"C\" fn timescaledb_toolkit_tdigest_free(_: Box<tdigest::TDigest>) {}\n\n// TODO Messy, but good enough to experiment with.  We might want to\n// into_raw_parts the String and offer a transparent struct containing pointer\n// to and size of the buffer, with a ts_tk_tdigest_string_free taking it back\n// and releasing it.  That also avoids one copy.\n#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn timescaledb_toolkit_tdigest_format_for_postgres(\n    td: *const tdigest::TDigest,\n) -> *mut libc::c_char {\n    let s = (*td).format_for_postgres();\n    let buf = libc::malloc(s.len() + 1);\n    libc::memcpy(buf, s.as_ptr() as *const libc::c_void, s.len());\n    let buf = buf as *mut libc::c_char;\n    let r = std::slice::from_raw_parts_mut(buf, s.len() + 1);\n    r[s.len()] = 0;\n    buf\n}\n"
  },
  {
    "path": "crates/time-weighted-average/Cargo.toml",
    "content": "[package]\nname = \"time_weighted_average\"\nversion = \"0.1.0\"\nauthors = [\"David Kohn <david@timescale.com>\"]\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nflat_serialize = {path=\"../flat_serialize/flat_serialize\"}\nflat_serialize_macro = {path=\"../flat_serialize/flat_serialize_macro\"}\nserde = { version = \"1.0\", features = [\"derive\"] }\ntspoint = {path=\"../tspoint\"}\n"
  },
  {
    "path": "crates/time-weighted-average/src/lib.rs",
    "content": "use serde::{Deserialize, Serialize};\nuse tspoint::TSPoint;\n\nuse flat_serialize_macro::FlatSerializable;\n\n#[derive(Clone, Copy, PartialEq, Eq, Debug, Serialize, Deserialize, FlatSerializable)]\n#[repr(u8)]\npub enum TimeWeightMethod {\n    LOCF = 0,\n    Linear,\n}\n\n#[derive(Clone, Copy, PartialEq, Debug, Serialize, Deserialize)]\npub struct TimeWeightSummary {\n    pub method: TimeWeightMethod,\n    pub first: TSPoint,\n    pub last: TSPoint,\n    pub w_sum: f64,\n}\n\n#[derive(PartialEq, Eq, Debug)]\npub enum TimeWeightError {\n    OrderError,\n    DoubleOverflow, // do we need to do this?\n    MethodMismatch,\n    InterpolateMissingPoint,\n    ZeroDuration,\n    EmptyIterator,\n}\n\nimpl TimeWeightSummary {\n    pub fn new(pt: TSPoint, method: TimeWeightMethod) -> Self {\n        TimeWeightSummary {\n            method,\n            first: pt,\n            last: pt,\n            w_sum: 0.0,\n        }\n    }\n\n    pub fn accum(&mut self, pt: TSPoint) -> Result<(), TimeWeightError> {\n        if pt.ts < self.last.ts {\n            return Err(TimeWeightError::OrderError);\n        }\n        if pt.ts == self.last.ts {\n            // if two points are equal we only use the first we see\n            // see discussion at https://github.com/timescale/timescaledb-toolkit/discussions/65\n            return Ok(());\n        }\n        self.w_sum += self.method.weighted_sum(self.last, pt);\n        self.last = pt;\n        Ok(())\n    }\n\n    // This combine function is different than some other combine functions as it requires disjoint time ranges in order to work\n    // correctly. The aggregate will never be parallel safe in the Postgres formulation because of this. However in the continuous\n    // aggregate context (and potentially in a multinode context) where we can be sure of disjoint time ranges, this will work.\n    // If there are space partitions, the space partition keys should be included in the group bys in order to be sure of this, otherwise\n    // overlapping ranges will be created.\n    pub fn combine(&self, next: &TimeWeightSummary) -> Result<TimeWeightSummary, TimeWeightError> {\n        if self.method != next.method {\n            return Err(TimeWeightError::MethodMismatch);\n        }\n        if self.last.ts >= next.first.ts {\n            // this combine function should always be pulling from disjoint sets, so duplicate values do not need to be handled\n            // as we do in accum() (where duplicates are ignored) here we throw an error, because duplicate values should\n            // always have been sorted into one or another bucket, and it means that the bounds of our buckets were wrong.\n            return Err(TimeWeightError::OrderError);\n        }\n        let new = TimeWeightSummary {\n            method: self.method,\n            first: self.first,\n            last: next.last,\n            w_sum: self.w_sum + next.w_sum + self.method.weighted_sum(self.last, next.first),\n        };\n        Ok(new)\n    }\n\n    pub fn new_from_sorted_iter<'a>(\n        iter: impl IntoIterator<Item = &'a TSPoint>,\n        method: TimeWeightMethod,\n    ) -> Result<TimeWeightSummary, TimeWeightError> {\n        let mut t = iter.into_iter();\n        let mut s = match t.next() {\n            None => {\n                return Err(TimeWeightError::EmptyIterator);\n            }\n            Some(val) => TimeWeightSummary::new(*val, method),\n        };\n        for p in t {\n            s.accum(*p)?;\n        }\n        Ok(s)\n    }\n\n    pub fn combine_sorted_iter<'a>(\n        iter: impl IntoIterator<Item = &'a TimeWeightSummary>,\n    ) -> Result<TimeWeightSummary, TimeWeightError> {\n        let mut t = iter.into_iter();\n        let mut s = match t.next() {\n            None => {\n                return Err(TimeWeightError::EmptyIterator);\n            }\n            Some(val) => *val,\n        };\n        for p in t {\n            s = s.combine(p)?;\n        }\n        Ok(s)\n    }\n\n    /// Extrapolate a TimeWeightSummary to bounds using the method and provided points outside the bounds of the original summary.\n    /// This is especially useful for cases where you want to get an average for, say, a time_bucket, using points outside of that time_bucket.\n    /// The initial aggregate will only have points within the time bucket, but outside of it, you will either have a point that you select\n    /// or a TimeWeightSummary where the first or last point can be used depending on which bound you are extrapolating to.\n    /// 1. The start_prev parameter is optional, but if a start is provided a previous point must be\n    ///    provided (for both linear and locf weighting methods).\n    /// 2. The end_next parameter is also optional, if an end is provided and the locf weighting\n    ///    method is specified, a next parameter isn't needed, with the linear method, the next\n    ///    point is needed and we will error if it is not provided.\n    pub fn with_bounds(\n        &self,\n        start_prev: Option<(i64, TSPoint)>,\n        end_next: Option<(i64, Option<TSPoint>)>,\n    ) -> Result<Self, TimeWeightError> {\n        let mut calc = *self;\n        if let Some((start, prev)) = start_prev {\n            calc = self.with_prev(start, prev)?\n        }\n\n        if let Some((end, next)) = end_next {\n            calc = self.with_next(end, next)?\n        }\n        Ok(calc)\n    }\n\n    fn with_prev(&self, target_start: i64, prev: TSPoint) -> Result<Self, TimeWeightError> {\n        // target_start must always be between [prev.ts, self.first.ts]\n        if prev.ts >= self.first.ts || target_start > self.first.ts || prev.ts > target_start {\n            return Err(TimeWeightError::OrderError); // should this be a different error?\n        }\n        if target_start == self.first.ts {\n            return Ok(*self);\n        }\n\n        let new_first = self\n            .method\n            .interpolate(prev, Some(self.first), target_start)?;\n        let w_sum = self.w_sum + self.method.weighted_sum(new_first, self.first);\n\n        Ok(TimeWeightSummary {\n            first: new_first,\n            w_sum,\n            ..*self\n        })\n    }\n\n    fn with_next(&self, target_end: i64, next: Option<TSPoint>) -> Result<Self, TimeWeightError> {\n        if target_end < self.last.ts {\n            // equal is okay, will just reduce to zero add in the sum, but not an error\n            return Err(TimeWeightError::OrderError);\n        }\n        // if our target matches last, there's no work to do, we're already there.\n        if target_end == self.last.ts {\n            return Ok(*self);\n        }\n\n        if let Some(next) = next {\n            if next.ts < target_end {\n                return Err(TimeWeightError::OrderError);\n            }\n        }\n\n        let new_last = self.method.interpolate(self.last, next, target_end)?;\n        let w_sum = self.w_sum + self.method.weighted_sum(self.last, new_last);\n\n        Ok(TimeWeightSummary {\n            last: new_last,\n            w_sum,\n            ..*self\n        })\n    }\n\n    ///Evaluate the time_weighted_average from the summary.\n    pub fn time_weighted_average(&self) -> Result<f64, TimeWeightError> {\n        if self.last.ts == self.first.ts {\n            return Err(TimeWeightError::ZeroDuration);\n        }\n        let duration = (self.last.ts - self.first.ts) as f64;\n        Ok(self.w_sum / duration)\n    }\n\n    /// Evaluate the integral in microseconds.\n    pub fn time_weighted_integral(&self) -> f64 {\n        if self.last.ts == self.first.ts {\n            // the integral of a duration of zero width is zero\n            0.0\n        } else {\n            self.w_sum\n        }\n    }\n}\n\nimpl TimeWeightMethod {\n    pub fn interpolate(\n        &self,\n        first: TSPoint,\n        second: Option<TSPoint>,\n        target: i64,\n    ) -> Result<TSPoint, TimeWeightError> {\n        if let Some(second) = second {\n            if second.ts <= first.ts {\n                return Err(TimeWeightError::OrderError);\n            }\n        }\n        let pt = TSPoint {\n            ts: target,\n            val: match (self, second) {\n                (TimeWeightMethod::LOCF, _) => first.val,\n                // TODO make this a method on TimeWeightMethod?\n                (TimeWeightMethod::Linear, Some(second)) => {\n                    first.interpolate_linear(&second, target).unwrap()\n                }\n                (TimeWeightMethod::Linear, None) => {\n                    return Err(TimeWeightError::InterpolateMissingPoint)\n                }\n            },\n        };\n        Ok(pt)\n    }\n\n    pub fn weighted_sum(&self, first: TSPoint, second: TSPoint) -> f64 {\n        debug_assert!(second.ts > first.ts);\n        let duration = (second.ts - first.ts) as f64;\n        match self {\n            TimeWeightMethod::LOCF => first.val * duration,\n            //the weighting for a linear interpolation is equivalent to the midpoint\n            //between the two values, this is because we're taking the area under the\n            //curve, which is the sum of the smaller of the two values multiplied by\n            //duration (a rectangle) + the triangle formed on top (abs diff between the\n            //two / 2 * duration) this is equivalent to the rectangle formed by the\n            //midpoint of the two.\n            //TODO: Stable midpoint calc? http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0811r2.html\n            TimeWeightMethod::Linear => (first.val + second.val) / 2.0 * duration,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::*;\n\n    // Just creating and basic use works\n    // Simple case gets correct results Done\n    // errors for each of with_prev/with_next,\n    // other error conditions:\n    // weird cases:\n    // NaN/Inf inputs -> should these error?\n    // Overflow? -> Inf\n    //\n\n    #[test]\n    fn test_simple_accum_locf() {\n        let mut s = TimeWeightSummary::new(TSPoint { ts: 0, val: 1.0 }, TimeWeightMethod::LOCF);\n        assert_eq!(s.w_sum, 0.0);\n        s.accum(TSPoint { ts: 10, val: 0.0 }).unwrap();\n        assert_eq!(s.w_sum, 10.0);\n        s.accum(TSPoint { ts: 20, val: 2.0 }).unwrap();\n        assert_eq!(s.w_sum, 10.0);\n        s.accum(TSPoint { ts: 30, val: 1.0 }).unwrap();\n        assert_eq!(s.w_sum, 30.0);\n        s.accum(TSPoint { ts: 40, val: -3.0 }).unwrap();\n        assert_eq!(s.w_sum, 40.0);\n        s.accum(TSPoint { ts: 50, val: -3.0 }).unwrap();\n        assert_eq!(s.w_sum, 10.0);\n    }\n    #[test]\n    fn test_simple_accum_linear() {\n        let mut s = TimeWeightSummary::new(TSPoint { ts: 0, val: 1.0 }, TimeWeightMethod::Linear);\n        assert_eq!(s.w_sum, 0.0);\n        s.accum(TSPoint { ts: 10, val: 0.0 }).unwrap();\n        assert_eq!(s.w_sum, 5.0);\n        s.accum(TSPoint { ts: 20, val: 2.0 }).unwrap();\n        assert_eq!(s.w_sum, 15.0);\n        s.accum(TSPoint { ts: 30, val: 1.0 }).unwrap();\n        assert_eq!(s.w_sum, 30.0);\n        s.accum(TSPoint { ts: 40, val: -3.0 }).unwrap();\n        assert_eq!(s.w_sum, 20.0);\n        s.accum(TSPoint { ts: 50, val: -3.0 }).unwrap();\n        assert_eq!(s.w_sum, -10.0);\n    }\n\n    fn new_from_sorted_iter_test(t: TimeWeightMethod) {\n        // simple test\n        let mut s = TimeWeightSummary::new(TSPoint { ts: 0, val: 1.0 }, t);\n        s.accum(TSPoint { ts: 10, val: 0.0 }).unwrap();\n        s.accum(TSPoint { ts: 20, val: 2.0 }).unwrap();\n        s.accum(TSPoint { ts: 30, val: 1.0 }).unwrap();\n\n        let n = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 0, val: 1.0 },\n                &TSPoint { ts: 10, val: 0.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 30, val: 1.0 },\n            ],\n            t,\n        )\n        .unwrap();\n        assert_eq!(s, n);\n\n        //single value\n        let s = TimeWeightSummary::new(TSPoint { ts: 0, val: 1.0 }, t);\n        let n =\n            TimeWeightSummary::new_from_sorted_iter(vec![&TSPoint { ts: 0, val: 1.0 }], t).unwrap();\n        assert_eq!(s, n);\n\n        //no values should error\n        let n = TimeWeightSummary::new_from_sorted_iter(vec![], t);\n        assert_eq!(n, Err(TimeWeightError::EmptyIterator));\n    }\n\n    #[test]\n    fn test_new_from_sorted_iter() {\n        new_from_sorted_iter_test(TimeWeightMethod::LOCF);\n        new_from_sorted_iter_test(TimeWeightMethod::Linear);\n    }\n\n    fn combine_test(t: TimeWeightMethod) {\n        let s = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 0, val: 1.0 },\n                &TSPoint { ts: 10, val: 0.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 30, val: 1.0 },\n            ],\n            t,\n        )\n        .unwrap();\n        let s1 = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 0, val: 1.0 }, &TSPoint { ts: 10, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n        let s2 = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 20, val: 2.0 }, &TSPoint { ts: 30, val: 1.0 }],\n            t,\n        )\n        .unwrap();\n        let s_comb = s1.combine(&s2).unwrap();\n        assert_eq!(s, s_comb);\n        // test combine with single val as well as multiple\n        let s21 = TimeWeightSummary::new(TSPoint { ts: 20, val: 2.0 }, t);\n        let s22 = TimeWeightSummary::new(TSPoint { ts: 30, val: 1.0 }, t);\n        assert_eq!(s1.combine(&s21).unwrap().combine(&s22).unwrap(), s);\n    }\n    #[test]\n    fn test_combine() {\n        combine_test(TimeWeightMethod::LOCF);\n        combine_test(TimeWeightMethod::Linear);\n    }\n\n    fn order_accum_test(t: TimeWeightMethod) {\n        let s = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 0, val: 1.0 }, &TSPoint { ts: 10, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n        let mut o = s;\n        // adding points at the same timestamp shouldn't affect the value (no matter whether the\n        // value is larger or smaller than the original)\n        o.accum(TSPoint { ts: 10, val: 2.0 }).unwrap();\n        assert_eq!(s, o);\n        o.accum(TSPoint { ts: 10, val: -1.0 }).unwrap();\n        assert_eq!(s, o);\n\n        //but adding out of order points doesn't work\n        assert_eq!(\n            o.accum(TSPoint { ts: 5, val: -1.0 }),\n            Err(TimeWeightError::OrderError)\n        );\n\n        //same for new_from_sorted_iter - test that multiple values only the first is taken\n        let n = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 0, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 30, val: 4.0 },\n            ],\n            t,\n        )\n        .unwrap();\n\n        let m = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 0, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 20, val: 0.0 },\n                &TSPoint { ts: 30, val: 4.0 },\n            ],\n            t,\n        )\n        .unwrap();\n        assert_eq!(m, n);\n\n        // but out of order inputs correctly error\n        let n = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 0, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 10, val: 0.0 },\n            ],\n            t,\n        );\n        assert_eq!(n, Err(TimeWeightError::OrderError));\n    }\n    #[test]\n    fn test_order_accum() {\n        order_accum_test(TimeWeightMethod::LOCF);\n        order_accum_test(TimeWeightMethod::Linear);\n    }\n\n    fn order_combine_test(t: TimeWeightMethod) {\n        let s = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 0, val: 1.0 }, &TSPoint { ts: 10, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n        let smaller = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 5, val: 1.0 }, &TSPoint { ts: 15, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n        // see note above, but\n        let equal = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 15, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n\n        assert_eq!(s.combine(&smaller), Err(TimeWeightError::OrderError));\n        assert_eq!(s.combine(&equal), Err(TimeWeightError::OrderError));\n    }\n    #[test]\n    fn test_order_combine() {\n        order_combine_test(TimeWeightMethod::LOCF);\n        order_combine_test(TimeWeightMethod::Linear);\n    }\n\n    fn combine_sorted_iter_test(t: TimeWeightMethod) {\n        //simple case\n        let m = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 0, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 30, val: 0.0 },\n                &TSPoint { ts: 40, val: 4.0 },\n            ],\n            t,\n        )\n        .unwrap();\n        let a = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 0, val: 1.0 }, &TSPoint { ts: 20, val: 2.0 }],\n            t,\n        )\n        .unwrap();\n        let b = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 30, val: 0.0 }, &TSPoint { ts: 40, val: 4.0 }],\n            t,\n        )\n        .unwrap();\n        let n = TimeWeightSummary::combine_sorted_iter(vec![&a, &b]).unwrap();\n        assert_eq!(m, n);\n\n        //single values are no problem\n        let n = TimeWeightSummary::combine_sorted_iter(vec![&m]).unwrap();\n        assert_eq!(m, n);\n\n        //single values in TimeWeightSummaries are no problem\n        let c = TimeWeightSummary::new(TSPoint { ts: 0, val: 1.0 }, t);\n        let d = TimeWeightSummary::new(TSPoint { ts: 20, val: 2.0 }, t);\n        let n = TimeWeightSummary::combine_sorted_iter(vec![&c, &d, &b]).unwrap();\n        assert_eq!(m, n);\n        // whether single values come first or later\n        let e = TimeWeightSummary::new(TSPoint { ts: 30, val: 0.0 }, t);\n        let f = TimeWeightSummary::new(TSPoint { ts: 40, val: 4.0 }, t);\n        let n = TimeWeightSummary::combine_sorted_iter(vec![&a, &e, &f]).unwrap();\n        assert_eq!(m, n);\n\n        // empty iterators error\n        assert_eq!(\n            TimeWeightSummary::combine_sorted_iter(vec![]),\n            Err(TimeWeightError::EmptyIterator)\n        );\n\n        // out of order values error\n        let n = TimeWeightSummary::combine_sorted_iter(vec![&c, &d, &f, &e]);\n        assert_eq!(n, Err(TimeWeightError::OrderError));\n\n        // even with two values\n        let n = TimeWeightSummary::combine_sorted_iter(vec![&b, &a]);\n        assert_eq!(n, Err(TimeWeightError::OrderError));\n    }\n    #[test]\n    fn test_combine_sorted_iter() {\n        combine_sorted_iter_test(TimeWeightMethod::LOCF);\n        combine_sorted_iter_test(TimeWeightMethod::Linear);\n    }\n\n    #[test]\n    fn test_mismatch_combine() {\n        let s1 = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 0, val: 1.0 }, &TSPoint { ts: 10, val: 0.0 }],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        let s2 = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 20, val: 2.0 }, &TSPoint { ts: 30, val: 1.0 }],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        assert_eq!(s1.combine(&s2), Err(TimeWeightError::MethodMismatch));\n\n        let s1 = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 0, val: 1.0 }, &TSPoint { ts: 10, val: 0.0 }],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        let s2 = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 20, val: 2.0 }, &TSPoint { ts: 30, val: 1.0 }],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        assert_eq!(s1.combine(&s2), Err(TimeWeightError::MethodMismatch));\n    }\n\n    #[test]\n    fn test_weighted_sum() {\n        let pt1 = TSPoint { ts: 10, val: 20.0 };\n        let pt2 = TSPoint { ts: 20, val: 40.0 };\n\n        let locf = TimeWeightMethod::LOCF.weighted_sum(pt1, pt2);\n        assert_eq!(locf, 200.0);\n\n        let linear = TimeWeightMethod::Linear.weighted_sum(pt1, pt2);\n        assert_eq!(linear, 300.0);\n\n        let pt2 = TSPoint { ts: 20, val: -40.0 };\n\n        let locf = TimeWeightMethod::LOCF.weighted_sum(pt1, pt2);\n        assert_eq!(locf, 200.0);\n\n        let linear = TimeWeightMethod::Linear.weighted_sum(pt1, pt2);\n        assert_eq!(linear, -100.0);\n    }\n\n    fn with_prev_common_test(t: TimeWeightMethod) {\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 20, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n        // target = starting point should produce itself no matter the method\n        let prev = TSPoint { ts: 5, val: 5.0 };\n        let target: i64 = 10;\n        assert_eq!(test.with_prev(target, prev).unwrap(), test);\n\n        // target = prev should always produce the same as if we made a new one with prev as the starting point, no matter the extrapolation method, though technically, this shouldn't come up in real world data, because you'd never target a place you had real data for, but that's fine, it's a useful reductive case for testing\n        let prev = TSPoint { ts: 5, val: 5.0 };\n        let target: i64 = 5;\n        let expected = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &prev,\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 0.0 },\n            ],\n            t,\n        )\n        .unwrap();\n        assert_eq!(test.with_prev(target, prev).unwrap(), expected);\n\n        // prev >= first should produce an order error\n        let prev = TSPoint { ts: 10, val: 5.0 };\n        let target: i64 = 10;\n        assert_eq!(\n            test.with_prev(target, prev).unwrap_err(),\n            TimeWeightError::OrderError\n        );\n\n        // target okay, but prev not less than it\n        let prev = TSPoint { ts: 5, val: 5.0 };\n        let target: i64 = 2;\n        assert_eq!(\n            test.with_prev(target, prev).unwrap_err(),\n            TimeWeightError::OrderError\n        );\n\n        // prev okay, but target > start\n        let prev = TSPoint { ts: 5, val: 5.0 };\n        let target: i64 = 15;\n        assert_eq!(\n            test.with_prev(target, prev).unwrap_err(),\n            TimeWeightError::OrderError\n        );\n    }\n\n    #[test]\n    fn test_with_prev() {\n        // adding a previous point is the same as a TimeWeightSummary constructed from the properly extrapolated previous point and the original\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 20, val: 0.0 }],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        let prev = TSPoint { ts: 0, val: 5.0 };\n        let target: i64 = 5;\n        let expected_origin = TSPoint { ts: 5, val: 5.0 };\n        let expected = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &expected_origin,\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 0.0 },\n            ],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        assert_eq!(test.with_prev(target, prev).unwrap(), expected);\n\n        // if the Summary uses a linear method, the extrapolation should be linear as well\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 20, val: 0.0 }],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        let prev = TSPoint { ts: 0, val: 5.0 };\n        let target: i64 = 5;\n        let expected_origin = TSPoint { ts: 5, val: 3.0 };\n        let expected = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &expected_origin,\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 0.0 },\n            ],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        assert_eq!(test.with_prev(target, prev).unwrap(), expected);\n\n        // now some common tests:\n        with_prev_common_test(TimeWeightMethod::Linear);\n        with_prev_common_test(TimeWeightMethod::LOCF);\n    }\n\n    fn with_next_common_test(t: TimeWeightMethod) {\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 20, val: 0.0 }],\n            t,\n        )\n        .unwrap();\n        // target = end point should produce itself no matter the method\n        let next = TSPoint { ts: 25, val: 5.0 };\n        let target: i64 = 20;\n        assert_eq!(test.with_next(target, Some(next)).unwrap(), test);\n\n        // target = next should always produce the same as if we added the next point for linear,  and will produce the same w_sum, though not the same final point for LOCF, here' we'll test the w_sum. Though technically, this shouldn't come up in real world data, because you'd never target a place you had real data for, but that's fine, it's a useful reductive case for testing\n        let next = TSPoint { ts: 25, val: 5.0 };\n        let target: i64 = 25;\n        let expected = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 0.0 },\n                &next,\n            ],\n            t,\n        )\n        .unwrap();\n        assert_eq!(\n            test.with_next(target, Some(next)).unwrap().w_sum,\n            expected.w_sum\n        );\n\n        // next <= last should produce an order error\n        let next = TSPoint { ts: 20, val: 5.0 };\n        let target: i64 = 22;\n        assert_eq!(\n            test.with_next(target, Some(next)).unwrap_err(),\n            TimeWeightError::OrderError\n        );\n\n        // target okay, but next not greater than it\n        let next = TSPoint { ts: 22, val: 5.0 };\n        let target: i64 = 25;\n        assert_eq!(\n            test.with_next(target, Some(next)).unwrap_err(),\n            TimeWeightError::OrderError\n        );\n\n        // next okay, but target < last\n        let next = TSPoint { ts: 25, val: 5.0 };\n        let target: i64 = 15;\n        assert_eq!(\n            test.with_next(target, Some(next)).unwrap_err(),\n            TimeWeightError::OrderError\n        );\n    }\n\n    #[test]\n    fn test_with_next() {\n        // adding a target_next point is the same as a TimeWeightSummary constructed from the properly extrapolated next point and the original\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 20, val: 2.0 }],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        let next = TSPoint { ts: 30, val: 3.0 };\n        let target: i64 = 25;\n        let expected_next = TSPoint { ts: 25, val: 2.0 };\n        let expected = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &expected_next,\n            ],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        assert_eq!(test.with_next(target, Some(next)).unwrap(), expected);\n        // For LOCF it doesn't matter if next is provided, only the target is required\n        assert_eq!(test.with_next(target, None).unwrap(), expected);\n\n        // if the Summary uses a linear method, the extrapolation should be linear as well\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![&TSPoint { ts: 10, val: 1.0 }, &TSPoint { ts: 20, val: 2.0 }],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        let next = TSPoint { ts: 30, val: 3.0 };\n        let target: i64 = 25;\n        let expected_next = TSPoint { ts: 25, val: 2.5 };\n        let expected = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &expected_next,\n            ],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        assert_eq!(test.with_next(target, Some(next)).unwrap(), expected);\n        // For Linear method, we need the second point, and not providing a next will error:\n        assert_eq!(\n            test.with_next(target, None).unwrap_err(),\n            TimeWeightError::InterpolateMissingPoint\n        );\n\n        // now some common tests:\n        with_next_common_test(TimeWeightMethod::Linear);\n        with_next_common_test(TimeWeightMethod::LOCF);\n    }\n\n    // add average tests\n    fn average_common_tests(t: TimeWeightMethod) {\n        let single = TimeWeightSummary::new(TSPoint { ts: 20, val: 2.0 }, t);\n        assert_eq!(\n            single.time_weighted_average().unwrap_err(),\n            TimeWeightError::ZeroDuration\n        );\n    }\n    #[test]\n    fn test_average() {\n        average_common_tests(TimeWeightMethod::Linear);\n        average_common_tests(TimeWeightMethod::LOCF);\n\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 30, val: 3.0 },\n            ],\n            TimeWeightMethod::LOCF,\n        )\n        .unwrap();\n        let expected = (10.0 * 1.0 + 10.0 * 2.0) / (30.0 - 10.0);\n        assert_eq!(test.time_weighted_average().unwrap(), expected);\n        let test = TimeWeightSummary::new_from_sorted_iter(\n            vec![\n                &TSPoint { ts: 10, val: 1.0 },\n                &TSPoint { ts: 20, val: 2.0 },\n                &TSPoint { ts: 30, val: 3.0 },\n            ],\n            TimeWeightMethod::Linear,\n        )\n        .unwrap();\n        let expected = (10.0 * 1.5 + 10.0 * 2.5) / (30.0 - 10.0);\n        assert_eq!(test.time_weighted_average().unwrap(), expected);\n    }\n}\n"
  },
  {
    "path": "crates/tspoint/Cargo.toml",
    "content": "[package]\nname = \"tspoint\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nflat_serialize = {path=\"../flat_serialize/flat_serialize\"}\nflat_serialize_macro = {path=\"../flat_serialize/flat_serialize_macro\"}\nserde = { version = \"1.0\", features = [\"derive\"] }\n"
  },
  {
    "path": "crates/tspoint/src/lib.rs",
    "content": "use serde::{ser::SerializeStruct, Deserialize, Serialize};\n\nuse flat_serialize_macro::FlatSerializable;\n\nuse std::ffi::CStr;\n\n#[derive(Clone, Copy, PartialEq, Debug, FlatSerializable)]\n#[repr(C)]\npub struct TSPoint {\n    pub ts: i64,\n    pub val: f64,\n}\n\n#[derive(Debug, PartialEq, Eq)]\npub enum TSPointError {\n    TimesEqualInterpolate,\n}\n\nimpl TSPoint {\n    pub fn interpolate_linear(&self, p2: &TSPoint, ts: i64) -> Result<f64, TSPointError> {\n        if self.ts == p2.ts {\n            return Err(TSPointError::TimesEqualInterpolate);\n        }\n        // using point slope form of a line iteratively y = y2 - y1 / (x2 - x1) * (x - x1) + y1\n        let duration = (p2.ts - self.ts) as f64; // x2 - x1\n        let dinterp = (ts - self.ts) as f64; // x - x1\n        Ok((p2.val - self.val) * dinterp / duration + self.val)\n    }\n}\n\nimpl Serialize for TSPoint {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        if serializer.is_human_readable() {\n            // FIXME ugly hack to use postgres functions in an non-postgres library\n            unsafe extern \"C\" {\n                fn _ts_toolkit_encode_timestamptz(dt: i64, buf: &mut [u8; 128]);\n            }\n            let mut ts = [0; 128];\n            unsafe {\n                _ts_toolkit_encode_timestamptz(self.ts, &mut ts);\n            }\n            let end = ts.iter().position(|c| *c == 0).unwrap();\n            let ts = CStr::from_bytes_with_nul(&ts[..end + 1]).unwrap();\n            let ts = ts.to_str().unwrap();\n            let mut point = serializer.serialize_struct(\"TSPoint\", 2)?;\n            point.serialize_field(\"ts\", &ts)?;\n            point.serialize_field(\"val\", &self.val)?;\n            point.end()\n        } else {\n            let mut point = serializer.serialize_struct(\"TSPoint\", 2)?;\n            point.serialize_field(\"ts\", &self.ts)?;\n            point.serialize_field(\"val\", &self.val)?;\n            point.end()\n        }\n    }\n}\n\nimpl<'de> Deserialize<'de> for TSPoint {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        use serde::de::{self, MapAccess, SeqAccess, Visitor};\n        use std::fmt;\n        struct TsPointVisitor {\n            text_timestamp: bool,\n        }\n\n        // FIXME ugly hack to use postgres functions in an non-postgres library\n        unsafe extern \"C\" {\n            // this is only going to be used to communicate with a rust lib we compile with this one\n            #[allow(improper_ctypes)]\n            fn _ts_toolkit_decode_timestamptz(text: &str) -> i64;\n        }\n\n        impl<'de> Visitor<'de> for TsPointVisitor {\n            type Value = TSPoint;\n\n            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {\n                formatter.write_str(\"struct TSPoint\")\n            }\n\n            fn visit_seq<V>(self, mut seq: V) -> Result<TSPoint, V::Error>\n            where\n                V: SeqAccess<'de>,\n            {\n                let ts = if self.text_timestamp {\n                    let text: &str = seq\n                        .next_element()?\n                        .ok_or_else(|| de::Error::invalid_length(0, &self))?;\n                    unsafe { _ts_toolkit_decode_timestamptz(text) }\n                } else {\n                    seq.next_element()?\n                        .ok_or_else(|| de::Error::invalid_length(0, &self))?\n                };\n                let val = seq\n                    .next_element()?\n                    .ok_or_else(|| de::Error::invalid_length(1, &self))?;\n                Ok(TSPoint { ts, val })\n            }\n\n            fn visit_map<V>(self, mut map: V) -> Result<TSPoint, V::Error>\n            where\n                V: MapAccess<'de>,\n            {\n                #[derive(Deserialize)]\n                #[serde(field_identifier, rename_all = \"lowercase\")]\n                enum Field {\n                    Ts,\n                    Val,\n                }\n                let mut ts = None;\n                let mut val = None;\n                while let Some(key) = map.next_key()? {\n                    match key {\n                        Field::Ts => {\n                            if ts.is_some() {\n                                return Err(de::Error::duplicate_field(\"ts\"));\n                            }\n                            ts = if self.text_timestamp {\n                                let text: &str = map.next_value()?;\n                                unsafe { Some(_ts_toolkit_decode_timestamptz(text)) }\n                            } else {\n                                Some(map.next_value()?)\n                            };\n                        }\n                        Field::Val => {\n                            if val.is_some() {\n                                return Err(de::Error::duplicate_field(\"val\"));\n                            }\n                            val = Some(map.next_value()?);\n                        }\n                    }\n                }\n                let ts = ts.ok_or_else(|| de::Error::missing_field(\"ts\"))?;\n                let val = val.ok_or_else(|| de::Error::missing_field(\"val\"))?;\n                Ok(TSPoint { ts, val })\n            }\n        }\n        const FIELDS: &[&str] = &[\"ts\", \"val\"];\n\n        let visitor = TsPointVisitor {\n            text_timestamp: deserializer.is_human_readable(),\n        };\n        deserializer.deserialize_struct(\"TSPoint\", FIELDS, visitor)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_linear_interpolate() {\n        let p1 = TSPoint { ts: 1, val: 1.0 };\n        let p2 = TSPoint { ts: 3, val: 3.0 };\n        assert_eq!(p1.interpolate_linear(&p2, 2).unwrap(), 2.0);\n        assert_eq!(p1.interpolate_linear(&p2, 3).unwrap(), 3.0);\n        assert_eq!(p1.interpolate_linear(&p2, 4).unwrap(), 4.0);\n        assert_eq!(p1.interpolate_linear(&p2, 0).unwrap(), 0.0);\n        assert_eq!(\n            p1.interpolate_linear(&p1, 2).unwrap_err(),\n            TSPointError::TimesEqualInterpolate\n        );\n    }\n}\n"
  },
  {
    "path": "crates/udd-sketch/Cargo.toml",
    "content": "[package]\nname = \"uddsketch\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nserde = { version = \"1.0\", features = [\"derive\"] }\n\n[dev-dependencies]\nordered-float = {version = \"1.0\", features = [\"serde\"] }\nrand = \"0.8.3\"\nquickcheck = \"1\"\nquickcheck_macros = \"1\"\n"
  },
  {
    "path": "crates/udd-sketch/src/lib.rs",
    "content": "//! UDDSketch implementation in rust.\n//! Based on the paper: https://arxiv.org/abs/2004.08604\n\nuse serde::{Deserialize, Serialize};\nuse std::collections::hash_map::Entry;\nuse std::collections::HashMap;\n\nuse crate::SketchHashKey::Invalid;\n#[cfg(test)]\nuse ordered_float::OrderedFloat;\n#[cfg(test)]\nuse std::collections::HashSet;\nuse std::num::NonZeroU32;\n\n#[cfg(test)]\nextern crate quickcheck;\n#[cfg(test)]\n#[macro_use(quickcheck)]\nextern crate quickcheck_macros;\n\n// This is used to index the buckets of the UddSketch.  In particular, because UddSketch stores values\n// based on a logarithmic scale, we need to track negative values separately from positive values, and\n// zero also needs special casing.\n#[derive(Serialize, Deserialize, Hash, PartialEq, Eq, Copy, Clone, Debug)]\npub enum SketchHashKey {\n    Negative(i64),\n    Zero,\n    Positive(i64),\n    Invalid,\n}\n\n// Invalid is treated as greater than valid values (making it a nice boundary value for list end)\nimpl std::cmp::Ord for SketchHashKey {\n    fn cmp(&self, other: &Self) -> std::cmp::Ordering {\n        use self::SketchHashKey::*;\n        use std::cmp::Ordering::*;\n\n        match (self, other) {\n            (Positive(a), Positive(b)) => a.cmp(b),\n            (Negative(a), Negative(b)) => a.cmp(b).reverse(),\n            (Positive(_), Negative(_) | Zero) => Greater,\n            (Negative(_) | Zero, Positive(_)) => Less,\n            (Zero, Negative(_)) => Greater,\n            (Negative(_), Zero) => Less,\n            (Zero, Zero) => Equal,\n            (Invalid, Invalid) => Equal,\n            (Invalid, Negative(_) | Zero | Positive(_)) => Greater,\n            (Negative(_) | Zero | Positive(_), Invalid) => Less,\n        }\n    }\n}\nimpl std::cmp::PartialOrd for SketchHashKey {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\n/// `UDDSketchMetadata` was created to avoid passing along many parameters\n/// to function calls.\npub struct UDDSketchMetadata {\n    pub max_buckets: u32,\n    pub current_error: f64,\n    pub compactions: u8,\n    pub values: u64,\n    pub sum: f64,\n    pub buckets: u32,\n}\nimpl SketchHashKey {\n    /// This is the key corresponding to the current key after the SketchHashMap it refers to has gone through one compaction.\n    /// Note that odd buckets get combined with the bucket after them (i.e. old buckets -3 and -2 become new bucket -1, {-1, 0} -> 0, {1, 2} -> 1)\n    fn compact_key(&self) -> SketchHashKey {\n        use SketchHashKey::*;\n\n        match *self {\n            Negative(i64::MAX) => *self, // Infinite buckets remain infinite\n            Positive(i64::MAX) => *self,\n            Negative(x) => Negative(if x > 0 { x + 1 } else { x } / 2),\n            Positive(x) => Positive(if x > 0 { x + 1 } else { x } / 2),\n            x => x, // Zero and Invalid don't compact\n        }\n    }\n}\n\n// Entries in the SketchHashMap contain a count and the next valid index of the map.\n#[derive(Debug, Clone, PartialEq)]\nstruct SketchHashEntry {\n    count: u64,\n    next: SketchHashKey,\n}\n\n// SketchHashMap is a special hash map of SketchHashKey->count that also keeps the equivalent of a linked list of the entries by increasing key value.\n#[derive(Debug, Clone, PartialEq)]\nstruct SketchHashMap {\n    map: HashMap<SketchHashKey, SketchHashEntry>,\n    head: SketchHashKey,\n}\n\nimpl std::ops::Index<SketchHashKey> for SketchHashMap {\n    type Output = u64;\n\n    fn index(&self, id: SketchHashKey) -> &Self::Output {\n        &self.map[&id].count\n    }\n}\n\n// Iterator for a SketchHashMap will travel through the map in order of increasing key value and return the (key, count) pairs\n#[derive(Clone)]\npub struct SketchHashIterator<'a> {\n    container: &'a SketchHashMap,\n    next_key: SketchHashKey,\n}\n\nimpl Iterator for SketchHashIterator<'_> {\n    type Item = (SketchHashKey, u64);\n\n    fn next(&mut self) -> Option<(SketchHashKey, u64)> {\n        if self.next_key == SketchHashKey::Invalid {\n            None\n        } else {\n            let key = self.next_key;\n            self.next_key = self.container.map[&self.next_key].next;\n            Some((key, self.container[key]))\n        }\n    }\n}\n\nimpl SketchHashMap {\n    fn new() -> SketchHashMap {\n        SketchHashMap {\n            map: HashMap::new(),\n            head: SketchHashKey::Invalid,\n        }\n    }\n\n    fn with_capacity(capacity: usize) -> SketchHashMap {\n        SketchHashMap {\n            map: HashMap::with_capacity(capacity),\n            head: SketchHashKey::Invalid,\n        }\n    }\n\n    /// Increment the count at a key, creating the entry if needed.\n    fn increment(&mut self, key: SketchHashKey) {\n        self.entry_upsert(key, 1);\n    }\n\n    fn iter(&self) -> SketchHashIterator<'_> {\n        SketchHashIterator {\n            container: self,\n            next_key: self.head,\n        }\n    }\n\n    /// Splits an entry if `key` is supposed to come right after it\n    /// Returns the key *after* the one that was split.\n    #[inline]\n    #[must_use] // The caller should really do something with this information.\n    fn entry_split(&mut self, key: SketchHashKey) -> SketchHashKey {\n        debug_assert_ne!(\n            key,\n            SketchHashKey::Invalid,\n            \"Invalid should never be used as a key into the SketchHashMap\"\n        );\n\n        let next: SketchHashKey;\n\n        // Special case, if we're actually in front of the Head,\n        // we're not really splitting the linked list, but prepending.\n        if key < self.head {\n            next = self.head;\n            self.head = key;\n            return next;\n        }\n\n        // Unfortunately, we'll now have to walk the whole map in order\n        // to find the location where we should be inserted\n        // into the single-linked list\n        for (k, e) in self.map.iter_mut() {\n            if *k < key && e.next > key {\n                next = e.next;\n                e.next = key;\n                return next;\n            }\n        }\n\n        unreachable!(\"Invalid key found\");\n    }\n\n    /// Upsert the given key/count into our map. This function\n    /// ensures the Linked List is in good shape afterwards.\n    #[inline]\n    fn entry_upsert(&mut self, key: SketchHashKey, count: u64) {\n        match self.map.entry(key) {\n            Entry::Occupied(mut o) => {\n                o.get_mut().count += count;\n                // Great, we don't have to update the Linked List\n                return;\n            }\n            Entry::Vacant(v) if self.head > key => {\n                v.insert(SketchHashEntry {\n                    count,\n                    next: self.head,\n                });\n                self.head = key;\n                // Great, we don't have to update the Linked List\n                return;\n            }\n            Entry::Vacant(_) => (), // We need to release our &mut map here, as we need to update 2 entries\n        };\n\n        // We've just inserted a new value, but need to ensure we fix the linked list again.\n        let new_next = self.entry_split(key);\n        self.map.insert(\n            key,\n            SketchHashEntry {\n                count,\n                next: new_next,\n            },\n        );\n    }\n\n    fn len(&self) -> usize {\n        self.map.len()\n    }\n\n    /// Combine adjacent buckets using the stack.\n    fn compact_using_stack<const N: usize>(&mut self) {\n        let len = self.map.len();\n        debug_assert!(len <= N);\n        let mut entries = [(SketchHashKey::Invalid, 0); N];\n        let mut drain = self.map.drain();\n\n        for e in entries.iter_mut() {\n            if let Some((key, entry)) = drain.next() {\n                *e = (key.compact_key(), entry.count);\n            } else {\n                break;\n            }\n        }\n        drop(drain);\n\n        self.populate_map_using_iter(&mut entries[0..len])\n    }\n\n    /// This function will populate the backing map using the provided slice.\n    /// It will sort and aggregate, so the caller does not need to take care\n    /// of that.\n    /// However, this should really only be called to populate the empty map.\n    fn populate_map_using_iter(&mut self, entries: &mut [(SketchHashKey, u64)]) {\n        assert!(\n            self.map.is_empty(),\n            \"SketchHashMap should be empty when populating using a slice\"\n        );\n        if entries.is_empty() {\n            return;\n        }\n\n        // To build up the linked list, we can do so by calling `entry_upsert` for every call\n        // to the `HashMap`. `entry_upsert` however needs to walk the map though to figure\n        // out where to place a key, therefore, we switch to:\n        // - sort\n        // - aggregate\n        // - insert\n        // That's what we do here\n\n        // - sort\n        entries.sort_unstable_by_key(|e| e.0);\n\n        // - aggregate\n        let mut old_index = 0;\n        let mut current = entries[0];\n        for idx in 1..entries.len() {\n            let next = entries[idx];\n            if next.0 == current.0 {\n                current.1 += next.1;\n            } else {\n                entries[old_index] = current;\n                current = next;\n                old_index += 1;\n            }\n        }\n\n        // Final one\n        entries[old_index] = current;\n\n        // We should only return the slice containing the aggregated values\n        let iter = entries.iter_mut().take(old_index + 1).peekable();\n\n        let mut iter = iter.peekable();\n        self.head = iter.peek().map(|p| p.0).unwrap_or(Invalid);\n\n        // - insert\n        while let Some((key, count)) = iter.next() {\n            self.map.insert(\n                *key,\n                SketchHashEntry {\n                    count: *count,\n                    next: iter.peek().map(|p| p.0).unwrap_or(Invalid),\n                },\n            );\n        }\n    }\n\n    #[inline]\n    fn compact(&mut self) {\n        match self.len() {\n            0 => (),\n            // PERCENTILE_AGG_DEFAULT_SIZE defaults to 200, so\n            // this entry covers that case.\n            1..=200 => self.compact_using_stack::<200>(),\n            201..=1000 => self.compact_using_stack::<1000>(),\n            1001..=5000 => self.compact_using_stack::<5000>(),\n            _ => self.compact_using_heap(),\n        }\n    }\n\n    // Combine adjacent buckets\n    fn compact_using_heap(&mut self) {\n        let mut entries = Vec::with_capacity(self.map.len());\n\n        // By draining the `HashMap`, we can reuse the same piece of memory after we're done.\n        // We're only using the `Vec` for a very short-lived period of time.\n        entries.extend(self.map.drain().map(|e| (e.0.compact_key(), e.1.count)));\n\n        self.populate_map_using_iter(&mut entries)\n    }\n}\n#[derive(Clone, Debug, PartialEq)]\npub struct UDDSketch {\n    buckets: SketchHashMap,\n    alpha: f64,\n    gamma: f64,\n    compactions: u8, // should always be smaller than 64\n    max_buckets: NonZeroU32,\n    num_values: u64,\n    values_sum: f64,\n}\n\nimpl UDDSketch {\n    pub fn new(max_buckets: u32, initial_error: f64) -> Self {\n        assert!((1e-12..1.0).contains(&initial_error));\n        UDDSketch {\n            buckets: SketchHashMap::new(),\n            alpha: initial_error,\n            gamma: (1.0 + initial_error) / (1.0 - initial_error),\n            compactions: 0,\n            max_buckets: NonZeroU32::new(max_buckets)\n                .expect(\"max buckets should be greater than zero\"),\n            num_values: 0,\n            values_sum: 0.0,\n        }\n    }\n\n    // This constructor is used to recreate a UddSketch from its component data\n    pub fn new_from_data(\n        metadata: &UDDSketchMetadata,\n        keys: impl Iterator<Item = SketchHashKey>,\n        mut counts: impl Iterator<Item = u64>,\n    ) -> Self {\n        let capacity = metadata.buckets as usize;\n        let mut sketch = UDDSketch {\n            buckets: SketchHashMap::with_capacity(capacity),\n            alpha: metadata.current_error,\n            gamma: gamma(metadata.current_error),\n            compactions: metadata.compactions,\n            max_buckets: NonZeroU32::new(metadata.max_buckets)\n                .expect(\"max buckets should be greater than zero\"),\n            num_values: metadata.values,\n            values_sum: metadata.sum,\n        };\n\n        let mut keys = keys.into_iter().peekable();\n        if let Some(key) = keys.peek() {\n            sketch.buckets.head = *key;\n        }\n\n        // This assumes the keys are unique and sorted\n        while let (Some(key), Some(count)) = (keys.next(), counts.next()) {\n            let next = keys.peek().copied().unwrap_or(Invalid);\n            sketch\n                .buckets\n                .map\n                .insert(key, SketchHashEntry { next, count });\n        }\n        debug_assert_eq!(sketch.buckets.map.len(), capacity);\n\n        sketch\n    }\n}\n\nimpl UDDSketch {\n    // For a given value return the index of it's bucket in the current sketch.\n    fn key(&self, value: f64) -> SketchHashKey {\n        key(value, self.gamma)\n    }\n\n    pub fn compact_buckets(&mut self) {\n        self.buckets.compact();\n\n        self.compactions += 1;\n        self.gamma *= self.gamma; // See https://arxiv.org/pdf/2004.08604.pdf Equation 3\n        self.alpha = 2.0 * self.alpha / (1.0 + self.alpha.powi(2)); // See https://arxiv.org/pdf/2004.08604.pdf Equation 4\n    }\n\n    pub fn bucket_iter(&self) -> SketchHashIterator<'_> {\n        self.buckets.iter()\n    }\n}\n\nimpl UDDSketch {\n    pub fn add_value(&mut self, value: f64) {\n        self.buckets.increment(self.key(value));\n\n        while self.buckets.len() > self.max_buckets.get() as usize {\n            self.compact_buckets();\n        }\n\n        self.num_values += 1;\n        self.values_sum += value;\n    }\n\n    /// `merge_items` will merge these values into the current sketch\n    /// it requires less memory than `merge_sketch`, as that needs a fully serialized\n    /// `UDDSketch`, whereas this function relies on iterators to do its job.\n    pub fn merge_items(\n        &mut self,\n        other: &UDDSketchMetadata,\n        mut keys: impl Iterator<Item = SketchHashKey>,\n        mut counts: impl Iterator<Item = u64>,\n    ) {\n        let other_gamma = gamma(other.current_error);\n        // Require matching initial parameters\n        debug_assert!(\n            (self\n                .gamma\n                .powf(1.0 / f64::powi(2.0, self.compactions as i32))\n                - other_gamma.powf(1.0 / f64::powi(2.0, other.compactions as i32)))\n            .abs()\n                < 1e-9 // f64::EPSILON too small, see issue #396\n        );\n        debug_assert_eq!(self.max_buckets.get(), other.max_buckets);\n\n        if other.values == 0 {\n            return;\n        }\n\n        while self.compactions < other.compactions {\n            self.compact_buckets();\n        }\n\n        let extra_compactions = self.compactions - other.compactions;\n        while let (Some(mut key), Some(count)) = (keys.next(), counts.next()) {\n            for _ in 0..extra_compactions {\n                key = key.compact_key();\n            }\n\n            self.buckets.entry_upsert(key, count);\n        }\n\n        while self.buckets.len() > self.max_buckets.get() as usize {\n            self.compact_buckets();\n        }\n\n        self.num_values += other.values;\n        self.values_sum += other.sum;\n    }\n\n    pub fn merge_sketch(&mut self, other: &UDDSketch) {\n        // Require matching initial parameters\n        assert!(\n            (self\n                .gamma\n                .powf(1.0 / f64::powi(2.0, self.compactions as i32))\n                - other\n                    .gamma\n                    .powf(1.0 / f64::powi(2.0, other.compactions as i32)))\n            .abs()\n                < 1e-9 // f64::EPSILON too small, see issue #396\n        );\n        assert!(self.max_buckets == other.max_buckets);\n\n        if other.num_values == 0 {\n            return;\n        }\n        if self.num_values == 0 {\n            *self = other.clone();\n            return;\n        }\n\n        let mut other = other.clone();\n\n        while self.compactions > other.compactions {\n            other.compact_buckets();\n        }\n        while other.compactions > self.compactions {\n            self.compact_buckets();\n        }\n\n        for entry in other.buckets.iter() {\n            let (key, value) = entry;\n            self.buckets.entry_upsert(key, value);\n        }\n\n        while self.buckets.len() > self.max_buckets.get() as usize {\n            self.compact_buckets();\n        }\n\n        self.num_values += other.num_values;\n        self.values_sum += other.values_sum;\n    }\n\n    pub fn max_allowed_buckets(&self) -> u32 {\n        self.max_buckets.get()\n    }\n\n    pub fn times_compacted(&self) -> u8 {\n        self.compactions\n    }\n\n    pub fn current_buckets_count(&self) -> usize {\n        self.buckets.map.len()\n    }\n}\n\nimpl UDDSketch {\n    #[inline]\n    pub fn mean(&self) -> f64 {\n        if self.num_values == 0 {\n            0.0\n        } else {\n            self.values_sum / self.num_values as f64\n        }\n    }\n\n    #[inline]\n    pub fn sum(&self) -> f64 {\n        self.values_sum\n    }\n\n    #[inline]\n    pub fn count(&self) -> u64 {\n        self.num_values\n    }\n\n    #[inline]\n    pub fn max_error(&self) -> f64 {\n        self.alpha\n    }\n\n    pub fn estimate_quantile(&self, quantile: f64) -> f64 {\n        estimate_quantile(\n            quantile,\n            self.alpha,\n            self.gamma,\n            self.num_values,\n            self.buckets.iter(),\n        )\n    }\n\n    pub fn estimate_quantile_at_value(&self, value: f64) -> f64 {\n        estimate_quantile_at_value(value, self.gamma, self.num_values, self.buckets.iter())\n    }\n}\n\npub fn estimate_quantile(\n    quantile: f64,\n    alpha: f64,\n    gamma: f64,\n    num_values: u64,\n    buckets: impl Iterator<Item = (SketchHashKey, u64)>,\n) -> f64 {\n    assert!((0.0..=1.0).contains(&quantile));\n\n    let mut remaining = (num_values as f64 * quantile) as u64 + 1;\n    if remaining >= num_values {\n        return last_bucket_value(alpha, gamma, buckets);\n    }\n\n    for entry in buckets {\n        let (key, count) = entry;\n        if remaining <= count {\n            return bucket_to_value(alpha, gamma, key);\n        } else {\n            remaining -= count;\n        }\n    }\n    unreachable!();\n}\n\n// Look up the value of the last bucket\n// This is not an efficient operation\nfn last_bucket_value(\n    alpha: f64,\n    gamma: f64,\n    buckets: impl Iterator<Item = (SketchHashKey, u64)>,\n) -> f64 {\n    let (key, _) = buckets.last().unwrap();\n    bucket_to_value(alpha, gamma, key)\n}\n\n/// inverse of `key()` within alpha\nfn bucket_to_value(alpha: f64, gamma: f64, bucket: SketchHashKey) -> f64 {\n    // When taking gamma ^ i below we have to use powf as powi only takes a u32, and i can exceed 2^32 for small alphas\n    match bucket {\n        SketchHashKey::Zero => 0.0,\n        SketchHashKey::Positive(i) => gamma.powf(i as f64 - 1.0) * (1.0 + alpha),\n        SketchHashKey::Negative(i) => -(gamma.powf(i as f64 - 1.0) * (1.0 + alpha)),\n        SketchHashKey::Invalid => panic!(\"Unable to convert invalid bucket id to value\"),\n    }\n}\n\npub fn estimate_quantile_at_value(\n    value: f64,\n    gamma: f64,\n    num_values: u64,\n    buckets: impl Iterator<Item = (SketchHashKey, u64)>,\n) -> f64 {\n    let mut count = 0.0;\n    let target = key(value, gamma);\n\n    for entry in buckets {\n        let (key, value) = entry;\n        if target > key {\n            count += value as f64;\n        } else {\n            if target == key {\n                // If the value falls in the target bucket, assume it's greater than half the other values\n                count += value as f64 / 2.0;\n            }\n            return count / num_values as f64;\n        }\n    }\n\n    1.0 // Greater than anything in the sketch\n}\n\nfn key(value: f64, gamma: f64) -> SketchHashKey {\n    let negative = value < 0.0;\n    let value = value.abs();\n\n    if value == 0.0 {\n        SketchHashKey::Zero\n    } else if negative {\n        SketchHashKey::Negative(value.log(gamma).ceil() as i64)\n    } else {\n        SketchHashKey::Positive(value.log(gamma).ceil() as i64)\n    }\n}\n\npub fn gamma(alpha: f64) -> f64 {\n    (1.0 + alpha) / (1.0 - alpha)\n}\n\n#[cfg(test)]\nmod tests {\n    use rand::{Rng, SeedableRng};\n\n    use super::*;\n\n    #[test]\n    fn build_and_add_values() {\n        let mut sketch = UDDSketch::new(20, 0.1);\n        sketch.add_value(1.0);\n        sketch.add_value(3.0);\n        sketch.add_value(0.5);\n\n        assert_eq!(sketch.count(), 3);\n        assert_eq!(sketch.mean(), 1.5);\n        assert_eq!(sketch.max_error(), 0.1);\n    }\n\n    #[test]\n    fn exceed_buckets() {\n        let mut sketch = UDDSketch::new(20, 0.1);\n        sketch.add_value(1.1); // Bucket #1\n        sketch.add_value(400.0); // Bucket #30\n        let a2 = 0.2 / 1.01;\n\n        assert_eq!(sketch.count(), 2);\n        assert_eq!(sketch.max_error(), 0.1);\n        for i in 2..20 {\n            sketch.add_value(1000.0 * 1.23_f64.powi(i));\n        }\n\n        assert_eq!(sketch.count(), 20);\n        assert_eq!(sketch.max_error(), 0.1);\n\n        for i in 20..30 {\n            sketch.add_value(1000.0 * 1.23_f64.powi(i));\n        }\n\n        assert_eq!(sketch.count(), 30);\n        assert_eq!(sketch.max_error(), a2);\n    }\n\n    /// We create this `merge_verifier` so that every test we run also tests\n    /// the multiple implementations we have for merging sketches.\n    /// It is a drop-in replacement for `merge_sketches`, with additional asserts.\n    fn merge_verifier(sketch: &mut UDDSketch, other: &UDDSketch) {\n        let mut second = sketch.clone();\n\n        sketch.merge_sketch(other);\n\n        let mut keys = Vec::with_capacity(other.num_values as usize);\n        let mut counts = Vec::with_capacity(other.num_values as usize);\n        for (key, count) in other.buckets.iter() {\n            keys.push(key);\n            counts.push(count);\n        }\n\n        let metadata = UDDSketchMetadata {\n            max_buckets: other.max_buckets.get(),\n            current_error: other.alpha,\n            compactions: other.compactions,\n            values: other.num_values,\n            sum: other.values_sum,\n            buckets: other.buckets.map.len() as u32,\n        };\n\n        second.merge_items(&metadata, keys.into_iter(), counts.into_iter());\n\n        // Both methods should result in the same end result.\n        assert_eq!(*sketch, second);\n    }\n\n    #[test]\n    fn merge_sketches() {\n        let a1 = 0.1; // alpha for up to 20 buckets\n        let a2 = 0.2 / 1.01; // alpha for 1 compaction\n        let a3 = 2.0 * a2 / (1.0 + f64::powi(a2, 2)); // alpha for 2 compactions\n        let a4 = 2.0 * a3 / (1.0 + f64::powi(a3, 2)); // alpha for 3 compactions\n        let a5 = 2.0 * a4 / (1.0 + f64::powi(a4, 2)); // alpha for 4 compactions\n\n        let mut sketch1 = UDDSketch::new(20, 0.1);\n        sketch1.add_value(1.1); // Bucket #1\n        sketch1.add_value(1.5); // Bucket #3\n        sketch1.add_value(1.6); // Bucket #3\n        sketch1.add_value(1.3); // Bucket #2\n        sketch1.add_value(4.2); // Bucket #8\n\n        assert_eq!(sketch1.count(), 5);\n        assert_eq!(sketch1.max_error(), a1);\n\n        let mut sketch2 = UDDSketch::new(20, 0.1);\n        sketch2.add_value(5.1); // Bucket #9\n        sketch2.add_value(7.5); // Bucket #11\n        sketch2.add_value(10.6); // Bucket #12\n        sketch2.add_value(9.3); // Bucket #12\n        sketch2.add_value(11.2); // Bucket #13\n\n        assert_eq!(sketch2.max_error(), a1);\n\n        merge_verifier(&mut sketch1, &sketch2);\n        assert_eq!(sketch1.count(), 10);\n        assert_eq!(sketch1.max_error(), a1);\n\n        let mut sketch3 = UDDSketch::new(20, 0.1);\n        sketch3.add_value(0.8); // Bucket #-1\n        sketch3.add_value(3.7); // Bucket #7\n        sketch3.add_value(15.2); // Bucket #14\n        sketch3.add_value(3.4); // Bucket #7\n        sketch3.add_value(0.6); // Bucket #-2\n\n        assert_eq!(sketch3.max_error(), a1);\n\n        merge_verifier(&mut sketch1, &sketch3);\n        assert_eq!(sketch1.count(), 15);\n        assert_eq!(sketch1.max_error(), a1);\n\n        let mut sketch4 = UDDSketch::new(20, 0.1);\n        sketch4.add_value(400.0); // Bucket #30\n        sketch4.add_value(0.004); // Bucket #-27\n        sketch4.add_value(0.0); // Zero Bucket\n        sketch4.add_value(-400.0); // Neg. Bucket #30\n        sketch4.add_value(-0.004); // Neg. Bucket #-27\n        sketch4.add_value(400000000000.0); // Some arbitrary large bucket\n        sketch4.add_value(0.00000005); // Some arbitrary small bucket\n        sketch4.add_value(-400000000000.0); // Some arbitrary large neg. bucket\n        sketch4.add_value(-0.00000005); // Some arbitrary small neg. bucket\n\n        assert_eq!(sketch4.max_error(), a1);\n\n        merge_verifier(&mut sketch1, &sketch4);\n\n        assert_eq!(sketch1.count(), 24);\n        assert_eq!(sketch1.max_error(), a2);\n\n        let mut sketch5 = UDDSketch::new(20, 0.1);\n        for i in 100..220 {\n            sketch5.add_value(1.23_f64.powi(i));\n        }\n\n        assert_eq!(sketch5.max_error(), a4);\n\n        merge_verifier(&mut sketch1, &sketch5);\n\n        assert_eq!(sketch1.count(), 144);\n        assert_eq!(sketch1.max_error(), a5); // Note that each compaction doesn't always result in half the numbers of buckets, hence a5 here instead of a4\n    }\n\n    #[test]\n    fn test_quantile_and_value_estimates() {\n        let mut sketch = UDDSketch::new(50, 0.1);\n        for v in 1..=10000 {\n            sketch.add_value(v as f64 / 100.0);\n        }\n        assert_eq!(sketch.count(), 10000);\n        assert_eq!(sketch.max_error(), 0.1);\n\n        for i in 1..=100 {\n            let value = i as f64;\n            let quantile = value / 100.0;\n            let quantile_value = value + 0.01; // correct value for quantile should be next number > value\n\n            let test_value = sketch.estimate_quantile(quantile);\n            let test_quant = sketch.estimate_quantile_at_value(value);\n\n            let percentage = (test_value - quantile_value).abs() / quantile_value;\n            assert!(\n                percentage <= 0.1,\n                \"Exceeded 10% error on quantile {}: expected {}, received {} (error% {})\",\n                quantile,\n                quantile_value,\n                test_value,\n                (test_value - quantile_value).abs() / quantile_value\n            );\n            let percentage = (test_quant - quantile).abs() / quantile;\n            assert!(\n                percentage < 0.2,\n                \"Exceeded 20% error on quantile at value {}: expected {}, received {} (error% {})\",\n                value,\n                quantile,\n                test_quant,\n                (test_quant - quantile).abs() / quantile\n            );\n        }\n\n        assert!((sketch.mean() - 50.005).abs() < 0.001);\n    }\n\n    #[test]\n    fn test_extreme_quantile_at_value() {\n        let mut sketch = UDDSketch::new(50, 0.1);\n        for v in 1..=10000 {\n            sketch.add_value(v as f64 / 100.0);\n        }\n\n        assert_eq!(sketch.estimate_quantile_at_value(-100.0), 0.0);\n        assert_eq!(sketch.estimate_quantile_at_value(0.0), 0.0);\n        assert_eq!(sketch.estimate_quantile_at_value(0.0001), 0.0);\n        assert_eq!(sketch.estimate_quantile_at_value(1000.0), 1.0);\n        assert!(sketch.estimate_quantile_at_value(0.01) < 0.0001);\n        assert!(sketch.estimate_quantile_at_value(100.0) > 0.9);\n    }\n\n    #[test]\n    fn random_stress() {\n        let mut sketch = UDDSketch::new(1000, 0.01);\n        let seed = rand::thread_rng().gen_range(0..u64::MAX);\n        let mut rng = rand::rngs::StdRng::seed_from_u64(seed);\n        let mut bounds = Vec::new();\n        for _ in 0..100 {\n            let v = rng.gen_range(-1000000.0..1000000.0);\n            sketch.add_value(v);\n            bounds.push(v);\n        }\n        bounds.sort_by(|a, b| a.partial_cmp(b).unwrap());\n\n        let mut prev = -2000000.0;\n        for f in bounds.iter() {\n            for _ in 0..10000 {\n                sketch.add_value(rng.gen_range(prev..*f));\n            }\n            prev = *f;\n        }\n\n        for i in 0..100 {\n            assert!(((sketch.estimate_quantile((i as f64 + 1.0) / 100.0) / bounds[i]) - 1.0).abs() < sketch.max_error() * bounds[i].abs(),\n                        \"Failed to correct match {} quantile with seed {}.  Received: {}, Expected: {}, Error: {}, Expected error bound: {}\",\n                        (i as f64 + 1.0) / 100.0,\n                        seed,\n                        sketch.estimate_quantile((i as f64 + 1.0) / 100.0),\n                        bounds[i],\n                        ((sketch.estimate_quantile((i as f64 + 1.0) / 100.0) / bounds[i]) - 1.0).abs() / bounds[i].abs(),\n                        sketch.max_error());\n        }\n    }\n\n    use crate::SketchHashKey::Invalid;\n    use quickcheck::*;\n\n    #[derive(Hash, PartialEq, Eq, PartialOrd, Ord, Copy, Clone, Debug)]\n    struct OrderedF64(OrderedFloat<f64>);\n\n    impl Arbitrary for OrderedF64 {\n        fn arbitrary(g: &mut Gen) -> Self {\n            OrderedF64(f64::arbitrary(g).into())\n        }\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_entry_invalid_hashmap_key() {\n        let mut map = SketchHashMap {\n            map: HashMap::new(),\n            head: Invalid,\n        };\n\n        map.entry_upsert(Invalid, 0);\n    }\n\n    #[test]\n    fn test_entry_insertion_order() {\n        let mut map = SketchHashMap {\n            map: HashMap::new(),\n            head: Invalid,\n        };\n\n        map.entry_upsert(SketchHashKey::Negative(i64::MIN), 5);\n        map.entry_upsert(SketchHashKey::Negative(10), 1);\n        map.entry_upsert(SketchHashKey::Positive(i64::MAX - 100), 17);\n        map.entry_upsert(SketchHashKey::Zero, 7);\n        map.entry_upsert(SketchHashKey::Positive(-10), 11);\n        map.entry_upsert(SketchHashKey::Negative(-10), 3);\n        map.entry_upsert(SketchHashKey::Positive(10), 13);\n\n        let keys: Vec<_> = map.iter().collect::<Vec<_>>();\n        assert_eq!(\n            keys,\n            vec![\n                (SketchHashKey::Negative(10), 1),\n                (SketchHashKey::Negative(-10), 3),\n                (SketchHashKey::Negative(i64::MIN), 5),\n                (SketchHashKey::Zero, 7),\n                (SketchHashKey::Positive(-10), 11),\n                (SketchHashKey::Positive(10), 13),\n                (SketchHashKey::Positive(i64::MAX - 100), 17),\n            ]\n        );\n\n        // We add some things before the current head, insert some new ones,\n        // add some to the end, and again inbetween some others\n        map.entry_upsert(SketchHashKey::Negative(i64::MAX), 3);\n        map.entry_upsert(SketchHashKey::Negative(-10), 23);\n        map.entry_upsert(SketchHashKey::Positive(9), 29);\n        map.entry_upsert(SketchHashKey::Positive(i64::MAX), 8);\n        map.entry_upsert(SketchHashKey::Positive(10), 123);\n        map.entry_upsert(SketchHashKey::Positive(11), 31);\n\n        let keys: Vec<_> = map.iter().collect::<Vec<_>>();\n        assert_eq!(\n            keys,\n            vec![\n                (SketchHashKey::Negative(i64::MAX), 3),\n                (SketchHashKey::Negative(10), 1),\n                (SketchHashKey::Negative(-10), 26), // 3 + 23\n                (SketchHashKey::Negative(i64::MIN), 5),\n                (SketchHashKey::Zero, 7),\n                (SketchHashKey::Positive(-10), 11),\n                (SketchHashKey::Positive(9), 29),\n                (SketchHashKey::Positive(10), 136), // 13 + 123\n                (SketchHashKey::Positive(11), 31),\n                (SketchHashKey::Positive(i64::MAX - 100), 17),\n                (SketchHashKey::Positive(i64::MAX), 8),\n            ]\n        );\n    }\n\n    #[quickcheck]\n    // Use multiple hashsets as input to allow a small number of duplicate values without getting ridiculous levels of duplication (as quickcheck is inclined to create)\n    fn fuzzing_test(\n        batch1: HashSet<OrderedF64>,\n        batch2: HashSet<OrderedF64>,\n        batch3: HashSet<OrderedF64>,\n        batch4: HashSet<OrderedF64>,\n    ) -> TestResult {\n        let mut master: Vec<f64> = batch1\n            .into_iter()\n            .chain(batch2.into_iter())\n            .chain(batch3.into_iter())\n            .chain(batch4.into_iter())\n            .map(|x| x.0.into())\n            .filter(|x: &f64| !x.is_nan())\n            .collect();\n\n        if master.len() < 100 {\n            return TestResult::discard();\n        }\n        let mut sketch = UDDSketch::new(100, 0.000001);\n        for value in &master {\n            sketch.add_value(*value);\n        }\n\n        let quantile_tests = [0.01, 0.1, 0.25, 0.5, 0.6, 0.8, 0.95];\n\n        master.sort_by(|a, b| a.partial_cmp(b).unwrap());\n\n        for i in 0..quantile_tests.len() {\n            let quantile = quantile_tests[i];\n\n            let mut test_val = sketch.estimate_quantile(quantile);\n\n            // If test_val is infinite, use the most extreme finite value to test relative error\n            if test_val.is_infinite() {\n                if test_val.is_sign_negative() {\n                    test_val = f64::MIN;\n                } else {\n                    test_val = f64::MAX;\n                }\n            }\n\n            // Compute target quantile using nearest rank definition\n            let master_idx = (quantile * master.len() as f64).floor() as usize;\n            let target = master[master_idx];\n            if target.is_infinite() {\n                continue; // trivially correct...or NaN, depending how you define it\n            }\n            let error = if target == 0.0 {\n                test_val\n            } else {\n                (test_val - target).abs() / target.abs()\n            };\n\n            assert!(error <= sketch.max_error(), \"sketch with error {} estimated {} quantile as {}, true value is {} resulting in relative error {}\n            values: {:?}\", sketch.max_error(), quantile, test_val, target, error, master);\n        }\n\n        TestResult::passed()\n    }\n}\n"
  },
  {
    "path": "docker/README.md",
    "content": "# Docker images\n\nTo speed up builds, we are using a set of pre-build docker images and\nthe Docker files for that is present in this directory.\n\n## Pre-requisites\n\nYou need to have Docker installed with support for DockerKit multi-platform\nand activate it by setting environment variable `DOCKER_BUILDKIT=1`.\n\n```bash\napt-get install docker.io\n```\n\n## Building multi-platform images\n\nTo build a new Docker image `toolkit-builder` for multiple platforms and push\nit to the development repository:\n\n```bash\nARCH=amd64\nOS_NAME=debian\nOS_VERSION=11\nOS_CODE_NAME=bullseye\nDOCKER_BUILDKIT=1 docker build \\\n                --platform $ARCH \\\n                --build-arg ARCH=$ARCH \\\n                --build-arg OS_NAME=$OS_NAME \\\n                --build-arg OS_VERSION=$OS_VERSION \\\n                --build-arg OS_CODE_NAME=$OS_CODE_NAME \\\n                -f docker/ci/Dockerfile \\\n                -t timescaledev/toolkit-builder-test:$OS_NAME-$OS_VERSION-$ARCH \\\n                .\n```\n\nWe publish the images as `timescaledev/toolkit-builder` instead of\n`timescaledev/toolkit-builder-test` after testing.\n\n## Troubleshooting\n\nIf you get the following error when pushing:\n\n```\n$ docker buildx build --platform linux/arm64/v8,linux/amd64 --tag timescaledev/toolkit-builder-test:latest --push .\n[+] Building 487.0s (54/54) FINISHED\n => [internal] load .dockerignore                                                                                                                                                                        0.0s\n => => transferring context: 2B                                                                                                                                                                          0.0s\n    .\n    .\n    .\n=> [auth] timescaledev/toolkit-builder-test:pull,push token for registry-1.docker.io                                                                                                                           0.0s\n------\n > exporting to image:\n------\nerror: failed to solve: failed to fetch oauth token: Post \"https://auth.docker.io/token\": x509: certificate has expired or is not yet valid: current time 2022-07-28T07:19:52+01:00 is after 2018-04-29T13:06:19Z\n```\n\nYou may have better luck with buildx instead of BuildKit.\nInstall from https://github.com/docker/buildx and then:\n\n```bash\nexport DOCKER_BUILDKIT=0\ndocker buildx build --platform linux/arm64/v8,linux/amd64 --tag timescaledev/toolkit-builder-test:latest --push .\n```\n"
  },
  {
    "path": "docker/ci/Dockerfile",
    "content": "ARG ARCH\nARG OS_NAME\nARG OS_VERSION\n\n# Without DockerKit, this doesn't work, even though documentation suggests it should.\n# With DockerKit, TARGETARCH is supposed to come in for free, but that doesn't work either.\nFROM --platform=${ARCH} ${OS_NAME}:${OS_VERSION} AS toolkit-base\n\nARG ARCH\nARG OS_CODE_NAME\n# Docker requires we redeclare these after FROM ¯\\_(ツ)_/¯\nARG OS_NAME\nARG OS_VERSION\n\nENV HOME=/home/postgres\n\n# Docker fails to set LOGNAME :(\nENV LOGNAME=root\nENV CARGO_HOME=/usr/local/cargo\nENV RUSTUP_HOME=/usr/local/rustup\nENV PATH=\"${CARGO_HOME}/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin\"\n\nCOPY docker/ci/setup.sh /\nCOPY tools/dependencies.sh /\n# TODO simple option processing a la build and testbin would make this less error-prone\nRUN /setup.sh ${ARCH} ${OS_NAME} ${OS_VERSION} \"${OS_CODE_NAME}\" postgres ${HOME}\n\n# TODO What does this 'AS' do?  It doesn't seem to name it.  We need -t for that.\nFROM toolkit-base AS toolkit-builder\n\nWORKDIR ${HOME}\n# Leave USER root for Github Actions.\n"
  },
  {
    "path": "docker/ci/setup.sh",
    "content": "#!/bin/sh\n\n# TODO rename to tools/setup - this is useful even for developer setup (add Mac/brew support)\n\nset -ex\n\nif [ -z \"$CARGO_HOME\" ] || [ -z \"$RUSTUP_HOME\" ]; then\n    echo >&2 'CARGO_HOME and RUSTUP_HOME environment variables must be set'\n    exit 3\nfi\n\nif [ \"$1\" = -unprivileged ]; then\n    privileged=false\n    shift\nelse\n    privileged=true\nfi\n\nif [ $# -ne 6 ]; then\n    echo >&2 'usage: setup.sh ARCH OS_NAME OS_VERSION OS_CODE_NAME BUILDER_USERNAME BUILDER_HOME'\n    exit 2\nfi\n\nARCH=$1\nOS_NAME=$2\nOS_VERSION=$3\nOS_CODE_NAME=$4\nBUILDER_USERNAME=$5\nBUILDER_HOME=$6\n\n. /dependencies.sh\n\n# Phase 0 - set platform-specific parameters\ncase $OS_NAME in\n    rockylinux)\n        PG_BASE=/usr/pgsql-\n        ;;\n    debian | ubuntu)\n        PG_BASE=/usr/lib/postgresql/\n        ;;\n    *)\n        echo >&2 \"unsupported $OS_NAME\"\n        exit 4\n        ;;\nesac\n\nif $privileged; then\n    # Phase 1 - cross-platform prerequisites\n    useradd -u 1001 -md \"$BUILDER_HOME\" $BUILDER_USERNAME\n\n    # Phase 2 - platform-specific package installation\n    case $OS_NAME in\n        # Red Hat Enterprise derivatives\n        rockylinux)\n            case $OS_VERSION in\n                8)\n                    yum -qy install dnf-plugins-core\n                    dnf config-manager --enable powertools\n                    dnf -qy module disable postgresql\n                    # fpm suddenly requires newer public_suffix that requires newer ruby\n                    # https://github.com/jordansissel/fpm/issues/1923 ¯\\_(ツ)_/¯\n                    dnf -qy module enable ruby:3.1\n                    dnf -qy install ruby-devel rubygems\n                    ;;\n\n                9)\n                    yum -qy install dnf-plugins-core\n                    dnf config-manager --enable crb\n                    dnf -qy install ruby-devel rubygems\n                    ;;\n\n                *)\n                    echo >&2 'only 7 - 9 supported'\n                    exit 5\n                    ;;\n            esac\n\n            # pgrx needs:\n            # - gcc (specifically; clang won't do!)\n            # - openssl-devel\n            # - make\n            # - pkg-config\n            yum -q -y install \\\n                gcc \\\n                git \\\n                make \\\n                openssl-devel \\\n                pkg-config \\\n                rpm-build \\\n                sudo\n\n            # Setup the postgresql.org package repository.\n            yum -q -y install https://download.postgresql.org/pub/repos/yum/reporpms/EL-${OS_VERSION}-${ARCH}/pgdg-redhat-repo-latest.noarch.rpm\n\n            # Setup the timescaledb package repository.\n            cat > /etc/yum.repos.d/timescale_timescaledb.repo <<EOF\n[timescale_timescaledb]\nname=timescale_timescaledb\nbaseurl=https://packagecloud.io/timescale/timescaledb/el/$OS_VERSION/$ARCH\nrepo_gpgcheck=1\ngpgcheck=0\nenabled=1\ngpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey\nsslverify=1\nsslcacert=/etc/pki/tls/certs/ca-bundle.crt\nmetadata_expire=300\nEOF\n\n            for pg in $PG_VERSIONS; do\n                yum -q -y install \\\n                    postgresql$pg-devel \\\n                    postgresql$pg-server\n                # We install as user postgres, so that needs write access to these.\n                chown $BUILDER_USERNAME $PG_BASE$pg/lib $PG_BASE$pg/share/extension\n            done;\n\n            for pg in $TSDB_PG_VERSIONS; do\n                yum -q -y install timescaledb-2-postgresql-$pg\n            done\n\n            gem install fpm -v $FPM_VERSION -N\n            ;;\n\n        # Debian family\n        debian | ubuntu)\n            # Image comes in with no package lists so we have to start with this.\n            apt-get -qq update\n\n            # Stop most debconf prompts.  Some have no default and we'd need\n            # to provide actual values via DEBCONF_OVERRIDE but we don't have\n            # any of those right now.\n            export DEBIAN_FRONTEND=noninteractive\n\n            # pgrx needs:\n            # - gcc (specifically; clang won't do!)\n            # - libssl-dev\n            # - make\n            # - pkg-config\n            # cmake and libkrb5-dev are needed to build TimescaleDB\n            apt-get -qq install \\\n                    build-essential \\\n                    cmake \\\n                    curl \\\n                    debhelper \\\n                    ed \\\n                    fakeroot \\\n                    gcc \\\n                    git \\\n                    gnupg \\\n                    libkrb5-dev \\\n                    libssl-dev \\\n                    make \\\n                    pkg-config \\\n                    postgresql-common \\\n                    sudo\n\n            # Setup the postgresql.org package repository.\n            # Don't use the -y flag as it is not supported on old versions of the script.\n            yes | /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh\n\n            # Setup the timescaledb package repository.\n            # TODO Blindly trusting a key that may change every time we run\n            #   defeats the purpose of package-signing but without a key rotation\n            #   story, the only security we have here is by trusting the system\n            #   certificate store, which we trust docker to provide a good copy\n            #   of.  May as well just put [trusted=yes] into sources.list instead\n            #   of bothering with apt-key...\n            if [ \"${OS_NAME}\" = \"ubuntu\" ]; then\n                curl -Ls https://packagecloud.io/timescale/timescaledb/gpgkey | apt-key add -\n            else\n                curl -Ls https://packagecloud.io/timescale/timescaledb/gpgkey | gpg --dearmor -o /etc/apt/trusted.gpg.d/timescale_timescaledb.gpg\n            fi\n\n            mkdir -p /etc/apt/sources.list.d\n            cat > /etc/apt/sources.list.d/timescaledb.list <<EOF\ndeb https://packagecloud.io/timescale/timescaledb/$OS_NAME/ $OS_CODE_NAME main\nEOF\n\n            apt-get -qq update\n\n            for pg in $PG_VERSIONS; do\n                apt-get -qq install \\\n                        postgresql-$pg \\\n                        postgresql-server-dev-$pg\n                # We install as user postgres, so that needs write access to these.\n                chown $BUILDER_USERNAME $PG_BASE$pg/lib /usr/share/postgresql/$pg/extension\n            done\n\n            for pg in $TSDB_PG_VERSIONS; do\n                # timescaledb packages Recommend toolkit, which we don't want here.\n                apt-get -qq install --no-install-recommends timescaledb-2-postgresql-$pg\n            done\n\n            # Ubuntu is the only system we want an image for that sticks an extra\n            # copy of the default PATH into PAM's /etc/environment and we su or sudo\n            # to $BUILDER_USERNAME thereby picking up that PATH and clobbering the\n            # one we set in Dockerfile.  There's nothing else in here, so at first I\n            # thought to remove it.  That works on 20.04 and 22.04, but still leaves\n            # a busted PATH on 18.04!  On 18.04, we get clobbered by ENV_PATH in\n            # /etc/login.defs .  We fix all three by setting our PATH here:\n            echo > /etc/environment \"PATH=$PATH\"\n            ;;\n    esac\n\n    # Phase 3 - cross-platform privileged tasks after package installation\n\n    # We've benefitted from being able to test expansions to our cargo\n    # installation without having to rebuild the CI image before, so\n    # donate the cargo installation to the builder user.\n    install -d -o $BUILDER_USERNAME \"$CARGO_HOME\" \"$RUSTUP_HOME\"\n\n    # We'll run tools/testbin as this user, and it needs to (un)install packages.\n    echo \"$BUILDER_USERNAME ALL=(ALL) NOPASSWD: ALL\" >> /etc/sudoers\n\n    cd \"$BUILDER_HOME\"\n    exec sudo -H --preserve-env=PATH,CARGO_HOME,RUSTUP_HOME -u $BUILDER_USERNAME \"$0\" -unprivileged \"$@\"\nfi\n\n# Phase 4 - unprivileged cross-platform tasks\n\ncurl -s https://sh.rustup.rs |\n    sh -s -- -q -y --no-modify-path --default-toolchain $RUST_TOOLCHAIN --profile $RUST_PROFILE -c $RUST_COMPONENTS\n\n# Install pgrx\ncargo install cargo-pgrx --version =$PGRX_VERSION\n\n# Configure pgrx\n## `cargo pgrx init` is not additive; must specify all versions in one command.\nfor pg_config in $(find /usr -name 'pg_config' -type f | grep \"${PG_BASE}\"); do\n    pg=\"$(${pg_config} --version | awk -F '[ .]' '{print $2'})\"\n    init_flags=\"$init_flags --pg$pg ${pg_config}\"\ndone\ncargo pgrx init $init_flags\n\n## Initialize pgrx-managed databases so we can add the timescaledb load, but only\n## for those PostgreSQL versions that have the timescaledb.so library available.\nfor pg_config in $(find /usr -name 'pg_config' -type f | grep \"${PG_BASE}\"); do\n    pg=\"$(${pg_config} --version | awk -F '[ .]' '{print $2'})\"\n    lib=\"$(find \"${PG_BASE}${pg}\" -type f -name 'timescaledb.so')\"\n    if [ \"${lib}\" != \"\" ]; then\n        echo \"shared_preload_libraries = 'timescaledb'\" >> ~/.pgrx/data-$pg/postgresql.conf\n    fi\ndone\n\n# Clone and fetch dependencies so we builds have less work to do.\ngit clone https://github.com/timescale/timescaledb-toolkit\ncd timescaledb-toolkit\ncargo fetch\n"
  },
  {
    "path": "docs/README.md",
    "content": "# TimescaleDB Toolkit Documentation\n---\nThe TimescaleDB Toolkit project contains a number of utilities for working with time-series data.  This documentation is further broken down by utility or feature in the list [below](#toolkit-features).\n\n## A note on tags <a id=\"tag-notes\"></a>\nFunctionality within the TimescaleDB Toolkit repository is intended to be introduced in varying stages of completeness.  To clarify which releases a given feature or function can be found in, the following tags are used:\n - **Experimental** - Denotes functionality that is still under very active development and may have poor performance, not handle corner cases or errors, etc.  Experimental APIs will change across releases, and extension-update will drop database objects that depend on experimental features. Do not use them unless you're willing to deal with the object you've created (the view, table, continuous aggregates, function, etc.) being dropped on update. This is particularly important for managed cloud services (like Timescale Cloud) that automate upgrades. Experimental features and functions can be found exclusively in the `toolkit_experimental` schema.\n - **Stable** ***release id*** - Functionality in this state should be correct and performant.  Stable APIs will be found in our releases and should not be broken in future releases.  Note that this tag will also be accompanied with the version in which the feature was originally released, such as: Feature Foo<sup><mark>stable-1.2</mark></sup>.\n - **Deprecated** - It may be necessary to remove stable functionality at some point, for instance if it is being supplanted by newer functionality or if it has deprecated dependencies.  Functionality with this tag is expected to be removed in future releases and current users of it should move to alternatives.\n\nNote that tags can be applied at either a feature or function scope.  The function tag takes precedence, but defaults to the feature scope if not present.  For example, if we have a feature `Foo` which is tagged `stable`, we would assume that an untagged function `FooCount` within that feature would be present in the current beta release.  However, if function `FooSum` were explicitly tagged `experimental` then we would only expect to find it in the nightly build.\n\n## Features <a id=\"toolkit-features\"></a>\n\nThe following links lead to pages for the different features in the TimescaleDB Toolkit repository.\n\n- [ASAP Smoothing](asap.md) [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes) - A data smoothing algorithm designed to generate human readable graphs which maintain any erratic data behavior while smoothing away the cyclic noise.\n- [Hyperloglog](hyperloglog.md) [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes) – An approximate `COUNT DISTINCT` based on hashing that provides reasonable accuracy in constant space. ([Methods](hyperloglog.md#hyperloglog_api))\n- [LTTB](lttb.md) [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes) – A downsample method that preserves visual similarity. ([Methods](lttb.md#api))\n\n- [Percentile Approximation](percentile_approximation.md) - A simple percentile approximation interface [([Methods](percentile_approximation.md#api))], wraps and simplifies the lower level algorithms:\n    - [T-Digest](tdigest.md) – A quantile estimate sketch optimized to provide more accurate estimates near the tails (i.e. 0.001 or 0.995) than conventional approaches. ([Methods](tdigest#tdigest_api))\n    - [UddSketch](uddsketch.md) – A quantile estimate sketch which provides a guaranteed maximum relative error. ([Methods](uddsketch.md#uddsketch_api))\n"
  },
  {
    "path": "docs/asap.md",
    "content": "# ASAP Smoothing [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes)\n\n> [Description](#asap-description)<br>\n> [Details](#asap-details)<br>\n> [Example](#asap-example)<br>\n> [API](#asap-api)\n\n## Description <a id=\"asap-description\"></a>\n\nThe [ASAP smoothing algorithm](https://arxiv.org/pdf/1703.00983.pdf) is designed create human readable graphs which preserve the rough shape and larger trends of the input data while minimizing the local variance between points.  TimescaleDB Toolkit provides an implementation of this which will take `(timestamp, value)` pairs, normalize them to the target interval, and return the ASAP smoothed values.\n\n## Details <a id=\"asap-details\"></a>\n\nTimescale's ASAP smoothing is implemented as a PostgresQL aggregate over a series of timestamps and values, with an additional target resolution used to control the output size.  The implementation will take the incoming data and attempt to bucket the points into even sized buckets such the number of buckets approximates the target resolution and each bucket contains a similar number of points (if necessary, gaps will be filled by interpolating the buckets on either side at this point).  It will then attempt to identify good candidate intervals for smoothing the data (using the Wiener-Khinchin theorem to find periods of high autocorrelation), and then choose the candidate that produces the smoothest graph while having the same degree of outlier values.\n\nThe output of the postgres aggregate is a timescale timevector object describing the start and step interval times and listing the values.  This can be passed to our `unnest` API to produce a table of time, value points.  The aggreates are also currently not partializeable or combinable.\n\n## Usage Example <a id=\"asap-example\"></a>\n\nIn this example we're going to examine about 250 years of monthly temperature readings from England (raw data can be found [here](http://futuredata.stanford.edu/asap/Temp.csv), though timestamps need to have a day added to be readable by PostgresQL).\n\n\n```SQL ,ignore\nCREATE TABLE temperatures(month TIMESTAMPTZ, value DOUBLE PRECISION);\nCOPY temperatures from 'temperature.csv' CSV HEADER;\nSELECT * FROM temperatures ORDER BY month LIMIT 10;\n```\n```\n            month             | value\n------------------------------+-------\n 1723-01-01 00:00:00-07:52:58 |   1.1\n 1723-02-01 00:00:00-07:52:58 |   4.4\n 1723-03-01 00:00:00-07:52:58 |   7.5\n 1723-04-01 00:00:00-07:52:58 |   8.9\n 1723-05-01 00:00:00-07:52:58 |  11.7\n 1723-06-01 00:00:00-07:52:58 |    15\n 1723-07-01 00:00:00-07:52:58 |  15.3\n 1723-08-01 00:00:00-07:52:58 |  15.6\n 1723-09-01 00:00:00-07:52:58 |  13.3\n 1723-10-01 00:00:00-07:52:58 |  11.1\n(10 rows)\n```\n\nIt is hard to look at this data and make much sense of how the temperature has changed over that time.  Here is a graph of the raw data:\n\n![Raw data](images/ASAP_raw.png)\n\nWe can use ASAP smoothing here to get a much clearer picture of the behavior over this interval.\n\n```SQL ,ignore\nSELECT * FROM unnest((SELECT asap_smooth(month, value, 800) FROM temperatures));\n```\n```\n                time                 |       value\n-------------------------------------+-------------------\n 1723-01-01 00:00:00-07:52:58        |  9.51550387596899\n 1723-04-12 21:38:55.135135-07:52:58 |   9.4890503875969\n 1723-07-23 19:17:50.27027-07:52:58  |  9.41656976744186\n 1723-11-02 16:56:45.405405-07:52:58 | 9.429360465116277\n 1724-02-12 14:35:40.54054-07:52:58  | 9.473546511627905\n 1724-05-24 12:14:35.675675-07:52:58 | 9.439341085271316\n 1724-09-03 09:53:30.81081-07:52:58  | 9.409496124031007\n 1724-12-14 07:32:25.945945-07:52:58 | 9.435465116279067\n 1725-03-26 05:11:21.08108-07:52:58  |  9.44864341085271\n 1725-07-06 02:50:16.216215-07:52:58 |  9.43003875968992\n 1725-10-16 00:29:11.35135-07:52:58  | 9.423062015503874\n 1726-01-25 22:08:06.486485-07:52:58 |  9.47771317829457\n 1726-05-07 19:47:01.62162-07:52:58  | 9.515310077519377\n 1726-08-17 17:25:56.756755-07:52:58 |  9.47383720930232\n...\n```\n\nNote the use of the `unnest` here to unpack the results of the `asap_smooth` command.  The output of this command is ~800 points of smoothed data (in this case it ended up being 888 points each representing a rolling moving average of about 21.5 years).  We can view of graph of these values to get a much clearer picture of how the temperature has fluctuated over this time:\n\n![Smoothed data](images/ASAP_smoothed.png)\n\n\n## Command List (A-Z) <a id=\"asap-api\"></a>\n> - [asap_smooth](#asap_smooth)\n\n---\n## **asap_smooth** <a id=\"asap_smooth\"></a>\n```SQL ,ignore\nasap_smooth(\n    ts TIMESTAMPTZ,\n    value DOUBLE PRECISION,\n    resolution INT\n) RETURNS NormalizedTimevector\n```\n\nThis normalize time, value pairs over a given interval and return a smoothed representation of those points.\n\n### Required Arguments <a id=\"asap-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `ts` | `TIMESTAMPTZ` | Column of timestamps corresponding to the values to aggregate |\n| `value` | `DOUBLE PRECISION` |  Column to aggregate. |\n| `resolution` | `INT` |  Approximate number of points to return.  Intended to represent the horizontal resolution in which the aggregate will be graphed\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `normalizedtimevector` | `NormalizedTimevector` | A object representing a series of values occurring at set intervals from a starting time.  It can be unpacked via `unnest` |\n<br>\n\n### Sample Usages <a id=\"asap-examples\"></a>\nFor this examples assume we have a table 'metrics' with columns 'date' and 'reading' which contains some interesting measurement we've accumulated over a large interval.  The following example would take that data and give us a smoothed representation of approximately 10 points which would still show any anomalous readings:\n\n<div hidden>\n\n```SQL ,non-transactional\nSET TIME ZONE 'UTC';\nCREATE TABLE metrics(date TIMESTAMPTZ, reading DOUBLE PRECISION);\nINSERT INTO metrics\nSELECT\n    '2020-1-1 UTC'::timestamptz + make_interval(hours=>foo),\n    (5 + 5 * sin(foo / 12.0 * PI()))\n    FROM generate_series(1,168) foo;\n\n```\n\n</div>\n\n```SQL\nSELECT time, round(value::numeric, 14) FROM unnest(\n    (SELECT asap_smooth(date, reading, 8)\n     FROM metrics));\n```\n```output\n             time             |       value\n------------------------------+-------------------\n       2020-01-01 01:00:00+00 | 5.18067120121489\n2020-01-02 00:51:25.714285+00 | 5.60453762172858\n 2020-01-03 00:42:51.42857+00 | 5.67427410239845\n2020-01-04 00:34:17.142855+00 | 5.34902995864025\n 2020-01-05 00:25:42.85714+00 | 4.81932879878511\n2020-01-06 00:17:08.571425+00 | 4.39546237827141\n 2020-01-07 00:08:34.28571+00 | 4.32572589760154\n2020-01-07 23:59:59.999995+00 | 4.65097004135974\n```"
  },
  {
    "path": "docs/client.md",
    "content": "# Client-side aggregation [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes)\n\n- Current status: prototype\n- Effort remaining: lots\n\n## Purpose\n\nWe have long suspected it might be valuable to allow building aggregates\nclient-side rather than requiring all data be stored in postgres and\naggregated within the toolkit.\n\nhttps://github.com/timescale/timescaledb-toolkit/issues/485 recently came in\nadding weight to this idea.  Because this customer requests tdigest, that's\nwhat we'll use for prototyping.\n\n## Use cases\n\nQuoting the above customer:\n\n\"In some cases it is not possible to transfer all the non-aggregated data to\nTimescaleDB due to it's amount and/or limited connectivity.\"\n\n## Questions\n\n- Do we want to support a public crate?\n  - What does that mean?\n  - Do we need to monitor an email address?\n  - What promise would we make on response time?\n  - Is this materially different from what we've already signed up for by\n    publishing on github?\n  - How do we handle ownership of the crates.io credentials?\n\n- Which license do we use?\n  - Some of our code is already a derived work - do we permissively license it\n    all, or restrict some of it?\n\n- Wire protocol maintenance\n  - This is a problem we already have, we just didn't realize it, as it is\n    already possible to construct our aggregates and INSERT them, and they\n    also in pg dumps; at the moment, you can restore those dumps, though we\n    haven't made any promise about it.  On our stabilized aggregates, users\n    may assume that is stabilized, too.\n  - Is there a practical concern here?  Or do we just say \"not supported\"?\n  - Is it possible to crash the extension with invalid inputs?\n  - If we commit to a public wire protocol, shouldn't we avoid the\n    Rust-specific ron and go for something more common?\n\n## Proposal\n\nAs a first step, build a crate which externalizes tdigest aggregate creation.\n\n```rust\nlet mut digester = tdigest::Builder::with_size(N);\nloop {\n    digester.push(value);\n}\nsend_to_postgres(format!(\"INSERT INTO digests VALUES ({})\", digester.build().format_for_postgres()));\n```\n\nIn order to provide that API, we must first reorganize the tdigest\nimplementation so that all business logic is in the tdigest crate.  Some is\ncurrently in the pgrx extension crate.\n\nFor each aggregate, the transient state is actually a Builder pattern hidden\nhidden behind pgrx machinery.\n\nOn this branch, I've moved TDigestTransState into tdigest::Builder.\n\nCurrently, we use default ron behavior to serialize the raw implementation\ndetails of the pg_type .  Users can insert inconsistent data now, and it\ndoesn't look like we validate that at insertion time.\n\nWe should reconsider this for all pg_types regardless of the overall client\nproject.  Is it possible NOT to offer serialized insertion at all?  If so,\nturning that off would be a good first step.\n\nThen we can enable it just where we want to.\n\nWe should put more thought into the serialization format we intentionally\nsupport.  Currently it contains redundancy which we can eliminate by\nimplementing serialization carefully rather than relying on defaults.\n\n## Proof of concept\n\nThis is a simple demonstration of inserting serialized tdigest into a table,\nshowing that it works the same way as an aggregate built by the extension.\n\n```SQL ,non-transactional\nCREATE TABLE test (data DOUBLE PRECISION);\nINSERT INTO test SELECT generate_series(0.01, 1, 0.01);\n\nCREATE VIEW digest AS SELECT tdigest(100, data) FROM test;\n\nCREATE TABLE digest2 (tdigest tdigest);\nINSERT INTO digest2 VALUES ('(version:1,max_buckets:100,count:100,sum:50.50000000000001,min:0.01,max:1,centroids:[(mean:0.01,weight:1),(mean:0.02,weight:1),(mean:0.03,weight:1),(mean:0.04,weight:1),(mean:0.05,weight:1),(mean:0.06,weight:1),(mean:0.07,weight:1),(mean:0.08,weight:1),(mean:0.09,weight:1),(mean:0.1,weight:1),(mean:0.11,weight:1),(mean:0.12,weight:1),(mean:0.13,weight:1),(mean:0.14,weight:1),(mean:0.15,weight:1),(mean:0.16,weight:1),(mean:0.17,weight:1),(mean:0.18,weight:1),(mean:0.19,weight:1),(mean:0.2,weight:1),(mean:0.21,weight:1),(mean:0.22,weight:1),(mean:0.23,weight:1),(mean:0.24,weight:1),(mean:0.25,weight:1),(mean:0.26,weight:1),(mean:0.27,weight:1),(mean:0.28,weight:1),(mean:0.29,weight:1),(mean:0.3,weight:1),(mean:0.31,weight:1),(mean:0.32,weight:1),(mean:0.33,weight:1),(mean:0.34,weight:1),(mean:0.35,weight:1),(mean:0.36,weight:1),(mean:0.37,weight:1),(mean:0.38,weight:1),(mean:0.39,weight:1),(mean:0.4,weight:1),(mean:0.41,weight:1),(mean:0.42,weight:1),(mean:0.43,weight:1),(mean:0.44,weight:1),(mean:0.45,weight:1),(mean:0.46,weight:1),(mean:0.47,weight:1),(mean:0.48,weight:1),(mean:0.49,weight:1),(mean:0.5,weight:1),(mean:0.51,weight:1),(mean:0.525,weight:2),(mean:0.545,weight:2),(mean:0.565,weight:2),(mean:0.585,weight:2),(mean:0.605,weight:2),(mean:0.625,weight:2),(mean:0.64,weight:1),(mean:0.655,weight:2),(mean:0.675,weight:2),(mean:0.69,weight:1),(mean:0.705,weight:2),(mean:0.72,weight:1),(mean:0.735,weight:2),(mean:0.75,weight:1),(mean:0.76,weight:1),(mean:0.775,weight:2),(mean:0.79,weight:1),(mean:0.8,weight:1),(mean:0.815,weight:2),(mean:0.83,weight:1),(mean:0.84,weight:1),(mean:0.85,weight:1),(mean:0.86,weight:1),(mean:0.87,weight:1),(mean:0.88,weight:1),(mean:0.89,weight:1),(mean:0.9,weight:1),(mean:0.91,weight:1),(mean:0.92,weight:1),(mean:0.93,weight:1),(mean:0.94,weight:1),(mean:0.95,weight:1),(mean:0.96,weight:1),(mean:0.97,weight:1),(mean:0.98,weight:1),(mean:0.99,weight:1),(mean:1,weight:1)])');\n```\n\n```SQL\nSELECT\n                    min_val(tdigest),\n                    max_val(tdigest),\n                    num_vals(tdigest)\n                    FROM digest;\n```\n```output\n min_val | max_val | num_vals\n---------+---------+----------\n    0.01 |       1 |      100\n```\n\nInserting serialized tdigest into table behaves the same:\n\n```SQL\nSELECT\n                    min_val(tdigest),\n                    max_val(tdigest),\n                    num_vals(tdigest)\n                    FROM digest2;\n```\n```output\n min_val | max_val | num_vals\n---------+---------+----------\n    0.01 |       1 |      100\n```\n"
  },
  {
    "path": "docs/counter_agg.md",
    "content": "# Counter Aggregates\n\n> [Description](#counter-agg-description)<br>\n> [Example Usage](#counter-agg-examples)<br>\n> [API](#counter-agg-api) <br>\n> [Notes on Parallelism and Ordering](#counter-agg-ordering)<br>\n> [Extrapolation Methods and Considerations](#counter-agg-methods)<br>\n\n\n## Description <a id=\"counter-agg-description\"></a>\n\nMetrics generally come in a few different varieties, which many systems have come to call *gauges* and *counters*. A gauge is a typical metric that can vary up or down, something like temperature or percent utilization. A counter is meant to be monotonically increasing. So it keeps track of, say, the total number of visitors to a website.\n\nThe main difference in processing counters and gauges is that a decrease in the value of a counter (compared to its previous value in the timevector) is interpreted as a *reset*. This means that the \"true value\" of the counter after a decrease is the previous value + the current value. A reset could occur due to a server restart or any number of other reasons. Because of the feature of the reset a counter is often analyzed by taking its change over a time period, accounting for resets. (Our `delta` function offers a way to do this).\n\nAccounting for resets is hard in pure SQL, so we've developed aggregate and accessor functions that do the proper calculations for counters. While the aggregate is not parallelizable, it is supported with [continuous aggregation](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).\n\nAdditionally, [see the notes on parallelism and ordering](#counter-agg-ordering) for a deeper dive into considerations for use with parallelism and some discussion of the internal data structures.\n\n---\n## Example Usage <a id=\"counter-agg-examples\"></a>\nFor these examples we'll assume a table `foo` defined as follows:\n```SQL ,ignore\nCREATE TABLE foo (\n    measure_id      BIGINT,\n    ts              TIMESTAMPTZ ,\n    val             DOUBLE PRECISION,\n    PRIMARY KEY (measure_id, ts)\n);\n```\n\nWe'll start by showing a typical usage of a counter aggregate as well as the `delta` accessor function which gives you the change in the counter's value over the time period in question, accounting for any resets.\n\n```SQL ,ignore\nSELECT measure_id,\n    delta(\n        counter_agg(ts, val)\n    )\nFROM foo\nGROUP BY measure_id;\n```\n\nWe can also use the [`time_bucket` function](https://docs.timescale.com/latest/api#time_bucket) to produce a series of deltas over 15 minute increments.\n```SQL ,ignore\nSELECT measure_id,\n    time_bucket('15 min'::interval, ts) as bucket,\n    delta(\n        counter_agg(ts, val)\n    )\nFROM foo\nGROUP BY measure_id, time_bucket('15 min'::interval, ts);\n```\n\nThis will allow us to search for 15 minute periods where the counter increased by a larger or smaller amount.\n\nIf series are less regular and so the deltas are affected by the number of samples in the 15 minute period, you can use the `extrapolated_delta` function. For this we'll need to provide bounds so we know where to extrapolate to, for this we'll use the `time_bucket_range` function, which works just like `time_bucket` but produces the open ended range `[start, end)` of all the times in the bucket. We'll also use a CTE to do the [`counter_agg`](#counter-agg-point) just so it's a little easier to understand what's going on in each part:\n\n```SQL ,ignore\nwith t as (\n    SELECT measure_id,\n        time_bucket('15 min'::interval, ts) as bucket,\n        counter_agg(ts, val, bounds => time_bucket_range('15 min'::interval, ts))\n    FROM foo\n    GROUP BY measure_id, time_bucket('15 min'::interval, ts))\nSELECT time_bucket,\n    extrapolated_delta(counter_agg, method => 'prometheus')\nFROM t ;\n```\n\nNote that we're also using the `'prometheus'` method for doing our extrapolation. Our current extrapolation function is built to mimic the Prometheus project's [`increase` function](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase), which measures the change of a counter extrapolated to the edges of the queried region.\n\nOf course this might be more useful if we make a continuous aggregate out of it. We'll first have to make it a hypertable partitioned on the ts column:\n\n```SQL ,ignore\nSELECT create_hypertable('foo', 'ts', chunk_time_interval=> '15 days'::interval, migrate_data => true);\n```\n\nNow we can make our continuous aggregate:\n\n```SQL ,ignore\nCREATE MATERIALIZED VIEW foo_15\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT measure_id,\n    time_bucket('15 min'::interval, ts) as bucket,\n    counter_agg(ts, val, bounds => time_bucket_range('15 min'::interval, ts))\nFROM foo\nGROUP BY measure_id, time_bucket('15 min'::interval, ts);\n```\n\nNote that here, we just use the [`counter_agg`](#counter-agg-point) function. It's often better to do that and simply run the accessor functions on the result, it's much more flexible that way, as there are many accessor functions, and the data is there so you can run multiple of them over the same aggregate.\n```SQL ,ignore\nSELECT\n    measure_id,\n    bucket,\n    delta(counter_agg),\n    rate(counter_agg),\n    extrapolated_rate(counter_agg, method => 'prometheus'),\n    slope(counter_agg)\nFROM foo_15\n```\n\nHere we've used multiple other accessor functions, the `rate` function is a simple `Δval / Δtime` (both observed) calculation, whereas the `extrapolated_rate` with the `'prometheus'` method follows the [Prometheus `rate` function's](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) behavior of extrapolating to the edges of the boundary and using the bounds provided rather than the observed values. The `slope` function calculates the slope of the least-squares fit line of the values over time. The counter resets are accounted for and \"true\" values are fed into the linear regression algorithm before this slope is computed.\n\nWe can also re-aggregate from the continuous aggregate into a larger bucket size quite simply:\n\n```SQL ,ignore\nSELECT\n    measure_id,\n    time_bucket('1 day'::interval, bucket),\n    delta(\n        rollup(counter_agg)\n    )\nFROM foo_15\nGROUP BY measure_id, time_bucket('1 day'::interval, bucket);\n```\n\nThere are several other accessor functions which we haven't described in the examples here, but are listed in the API section under the [accessors](#counter-agg-api-accessors).\n\n---\n\n# Command List  <a id=\"counter-agg-api\"></a>\n\n### [Aggregate Functions](#counter-agg-api-aggregates)\n> - [counter_agg() (point form)](#counter-agg-point)\n> - [rollup() (summary form)](#counter-agg-summary)\n### [Accessor Functions (A-Z)](#counter-agg-api-accessors)\n> - [corr()](#counter-agg-corr)\n> - [counter_zero_time()](#counter-agg-counter-zero-time)\n> - [delta()](#counter-agg-delta)\n> - [extrapolated_delta()](#counter-agg-extrapolated-delta)\n> - [extrapolated_rate()](#counter-agg-extrapolated-rate)\n> - [idelta_left()](#counter-agg-idelta-left)\n> - [idelta_right()](#counter-agg-idelta-right)\n> - [intercept()](#counter-agg-intercept)\n> - [irate_left()](#counter-agg-irate-left)\n> - [irate_right()](#counter-agg-irate-right)\n> - [num_changes()](#counter-agg-num-changes)\n> - [num_elements()](#counter-agg-num-elements)\n> - [num_resets()](#counter-agg-num-resets)\n> - [rate()](#counter-agg-rate)\n> - [slope()](#counter-agg-slope)\n> - [time_delta()](#counter-agg-time-delta)\n### [Utility Functions](#counter-agg-api-utilities)\n> - [with_bounds()](#counter-agg-with-bounds)\n---\n\n\n# Aggregate Functions <a id=\"counter-agg-api-aggregates\"></a>\nAggregating a counter to produce a `CounterSummary` is the first step in performing any calculations on it. There are two basic forms, one which takes in timestamps and values (the point form) and one which can combine multiple `CounterSummaries` together to form a larger summary spanning a larger amount of time. (See [Notes on Parallelism and Ordering](#counter-agg-ordering) for more information on how that works).\n\n---\n## **counter_agg() (point form)** <a id=\"counter-agg-point\"></a>\n```SQL ,ignore\ncounter_agg(\n    ts TIMESTAMPTZ,\n    value DOUBLE PRECISION¹,\n    bounds TSTZRANGE DEFAULT NULL\n) RETURNS CounterSummary\n```\n\nAn aggregate that produces a `CounterSummary` from timestamps and associated values.\n\n##### ¹ Note that the `value` is currently only accepted as a `DOUBLE PRECISION` number as most people use that for counters, even though other numeric types (ie `BIGINT`) might sometimes be more intuitive. If you store a value as a different numeric type you can cast to `DOUBLE PRECISION` on input to the function.\n\n### Required Arguments²\n|Name| Type |Description|\n|---|---|---|\n| `ts` | `TIMESTAMPTZ` |  The time at each point |\n| `value` | `DOUBLE PRECISION` | The value at each point to use for the counter aggregate|\n<br>\n\n##### ² Note that `ts` and `value` can be `null`, however the aggregate is not evaluated on `null` values and will return `null`, but it will not error on `null` inputs.\n\n### Optional Arguments\n|Name| Type |Description|\n|---|---|---|\n| `bounds` | `TSTZRANGE` |  A range of `timestamptz` representing the largest and smallest possible times that could be input to this aggregate. Calling with `NULL` or leaving out the argument results in an unbounded `CounterSummary`. Bounds are required for extrapolation, but not for other [accessor functions](#counter-agg-api-accessors). |\n\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `counter_agg` | `CounterSummary` | A CounterSummary object that can be passed to [accessor functions](#counter-agg-api-accessors) or other objects in the counter aggregate API |\n<br>\n\n### Sample Usage\n```SQL ,ignore\nWITH t as (\n    SELECT\n        time_bucket('1 day'::interval, ts) as dt,\n        counter_agg(ts, val) AS cs -- get a CounterSummary\n    FROM foo\n    WHERE id = 'bar'\n    GROUP BY time_bucket('1 day'::interval, ts)\n)\nSELECT\n    dt,\n    irate_right(cs) -- extract instantaneous rate from the CounterSummary\nFROM t;\n```\n\n---\n## **rollup() (summary form)**<a id=\"counter-agg-summary\"></a>\n```SQL ,ignore\nrollup(\n    cs CounterSummary\n) RETURNS CounterSummary\n```\n\nAn aggregate to compute a combined `CounterSummary` from a series of non-overlapping `CounterSummaries`. Non-disjoint `CounterSummaries` will cause errors. See [Notes on Parallelism and Ordering](#counter-agg-ordering) for more information.\n\n### Required Arguments²\n|Name| Type |Description|\n|---|---|---|\n| `cs` | `CounterSummary` | The input CounterSummary from a previous [`counter_agg`](#counter-agg-point) (point form) call, often from a [continuous aggregate](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates)|\n\n##### ² Note that `summary` can be `null`, however the aggregate is not evaluated on `null` values and will return `null`, but it will not error on `null` inputs.\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `counter_agg` | `CounterSummary` |  A CounterSummary object that can be passed to [accessor functions](#counter-agg-api-accessors) or other objects in the counter aggregate API|\n<br>\n\n### Sample Usage\n```SQL ,ignore\nWITH t as (\n    SELECT\n        date_trunc('day', ts) as dt,\n        counter_agg(ts, val) AS counter_summary -- get a time weight summary\n    FROM foo\n    WHERE id = 'bar'\n    GROUP BY date_trunc('day')\n), q as (\n    SELECT rollup(counter_summary) AS full_cs -- do a second level of aggregation to get the full CounterSummary\n    FROM t\n)\nSELECT\n    dt,\n    delta(counter_summary),  -- extract the delta from the  CounterSummary\n    delta(counter_summary) / (SELECT delta(full_cs) FROM q LIMIT 1)  as normalized -- get the fraction of the delta that happened each day compared to the full change of the counter\nFROM t;\n```\n# Accessor Functions <a id=\"counter-agg-api-accessors\"></a>\n\n## Accessor Function List (by family)\n### [Change over time (delta) functions](#counter-agg-delta-fam)\n> - [delta()](#counter-agg-delta)\n> - [extrapolated_delta()](#counter-agg-extrapolated-delta)\n> - [idelta_left()](#counter-agg-idelta-left)\n> - [idelta_right()](#counter-agg-idelta-right)\n> - [time_delta()](#counter-agg-time-delta)\n\n### Rate of change over time (rate) functions\n> - [rate()](#counter-agg-rate)\n> - [extrapolated_rate()](#counter-agg-extrapolated-rate)\n> - [irate_left()](#counter-agg-irate-left)\n> - [irate_right()](#counter-agg-irate-right)\n\n### Counting functions\n> - [num_changes()](#counter-agg-num-changes)\n> - [num_elements()](#counter-agg-num-elements)\n> - [num_resets()](#counter-agg-num-resets)\n\n### Statistical regression / least squares fit functions\n> - [slope()](#counter-agg-slope)\n> - [intercept()](#counter-agg-intercept)\n> - [counter_zero_time()](#counter-agg-counter-zero-time)\n> - [corr()](#counter-agg-corr)\n\n\n\n---\n## **Change over time (delta) functions** <a id=\"counter-agg-delta-fam\"></a>\nFunctions in the delta family are dedicated to finding the change in a value (or observed time, in the case of `time_delta`) of a counter during a time period, taking into account any counter resets that may have occurred.\n\n---\n## **delta()** <a id=\"counter-agg-delta\"></a>\n```SQL ,ignore\ndelta(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\nThe change in the counter over the time period. This is the raw or simple delta computed by accounting for resets then subtracting the last seen value from the first.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `delta` | `DOUBLE PRECISION` | The delta computed from the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-delta-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    delta(summary)\nFROM (\n    SELECT\n        id,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id\n) t\n```\n\n---\n## **extrapolated_delta()** <a id=\"counter-agg-extrapolated-delta\"></a>\n```SQL ,ignore\nextrapolated_delta(\n    summary CounterSummary,\n    method TEXT¹\n) RETURNS DOUBLE PRECISION\n```\nThe change in the counter during the time period specified by the `bounds` in the `CounterSummary`. To calculate the extrapolated delta, any counter resets are accounted for and the observed values are extrapolated to the bounds using the `method` specified (see [Extrapolation Methods and Considerations](#counter-agg-methods)) then the values are subtracted to compute the delta.\n\nThe `bounds` must be specified for the `extrapolated_delta` function to work, the bounds can be provided in the [`counter_agg`](#counter-agg-point) call, or by using the [`with_bounds`](#counter-agg-with-bounds) utility function to set the bounds\n\n##### ¹ Currently, the only allowed value of `method` is `'prometheus'`, as we have only implemented extrapolation following the Prometheus extrapolation protocol, see [Extrapolation Methods and Considerations](#counter-agg-methods) for more information.\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n| `method` | `TEXT` | The extrapolation method to use, the only option currently is 'prometheus', not case sensitive.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `extrapolated_delta` | `DOUBLE PRECISION` | The delta computed from the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-extrapolated-delta-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    extrapolated_delta(\n        with_bounds(\n            summary,\n            time_bucket_range('15 min'::interval, bucket)\n        )\n    )\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **idelta_left()** <a id=\"counter-agg-idelta-left\"></a>\n```SQL ,ignore\nidelta_left(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe instantaneous change in the counter at the left (earlier) side of the time range. Essentially, the first value subtracted from the second value seen in the time range (handling resets appropriately). This can be especially useful for fast moving counters.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `idelta_left` | `DOUBLE PRECISION` | The instantaneous delta computed from left (earlier) side of the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-idelta_left-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    idelta_left(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **idelta_right()** <a id=\"counter-agg-idelta-right\"></a>\n```SQL ,ignore\nidelta_right(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe instantaneous change in the counter at the right (later) side of the time range. Essentially, the penultimate value subtracted from the last value seen in the time range (handling resets appropriately). This can be especially useful for fast moving counters.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `idelta_right` | `DOUBLE PRECISION` | The instantaneous delta computed from right (later) side of the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-idelta_left-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    idelta_right(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **time_delta()** <a id=\"counter-agg-time-delta\"></a>\n```SQL ,ignore\ntime_delta(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe observed change in time (`last time - first time`) over the period aggregated. Measured in seconds.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point)  call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `time_delta` | `DOUBLE PRECISION` | The total duration in seconds between the first and last observed times in the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-time-delta-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    time_delta(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **Rate of change over time (rate) functions** <a id=\"counter-agg-rate-fam\"></a>\nThe rate family of functions find the reset-adjusted rate of change (`delta(value)/delta(time)`) of a counter on a per-second basis.\n\n---\n## **rate()** <a id=\"counter-agg-rate\"></a>\n```SQL ,ignore\nrate(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\nThe rate of change of the counter over the observed time period.  This is the raw or simple rate, equivalent to `delta(summary) / time_delta(summary)`. After accounting for resets, we subtract the last value from the first and divide by the duration between the last observed time and the first observed time.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `rate` | `DOUBLE PRECISION` | The per second observed rate computed from the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-rate-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    rate(summary)\nFROM (\n    SELECT\n        id,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id\n) t\n```\n\n---\n## **extrapolated_rate()** <a id=\"counter-agg-extrapolated-rate\"></a>\n```SQL ,ignore\nextrapolated_rate(\n    summary CounterSummary,\n    method TEXT¹\n) RETURNS DOUBLE PRECISION\n```\nThe rate of change in the counter computed over the time period specified by the `bounds` in the `CounterSummary`, extrapolating to the edges. Essentially, it is an [`extrapolated_delta`](#counter-agg-extrapolated-delta) divided by the duration in seconds.\n\nThe `bounds` must be specified for the `extrapolated_rate` function to work, the bounds can be provided in the [`counter_agg`](#counter-agg-point) call, or by using the [`with_bounds`](#counter-agg-with-bounds) utility function to set the bounds\n\n##### ¹ Currently, the only allowed value of `method` is `'prometheus'`, as we have only implemented extrapolation following the Prometheus extrapolation protocol, see [Extrapolation Methods and Considerations](#counter-agg-methods) for more information.\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n| `method` | `TEXT` | The extrapolation method to use, the only option currently is 'prometheus', not case sensitive.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `extrapolated_rate` | `DOUBLE PRECISION` | The per-second rate of change of the counter computed from the `CounterSummary` extrapolated to the `bounds` specified there. |\n<br>\n\n### Sample Usage <a id=\"counter-agg-extrapolated-rate-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    extrapolated_rate(\n        with_bounds(\n            summary,\n            time_bucket_range('15 min'::interval, bucket)\n        )\n    )\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **irate_left()** <a id=\"counter-agg-irate-left\"></a>\n```SQL ,ignore\nirate_left(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe instantaneous rate of change of the counter at the left (earlier) side of the time range. Essentially, the [`idelta_left`](#counter-agg-idelta-left) divided by the duration between the first and second observed points in the `CounterSummary`. This can be especially useful for fast moving counters.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `irate_left` | `DOUBLE PRECISION` | The instantaneous rate computed from left (earlier) side of the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-irate-left-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    irate_left(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **irate_right()** <a id=\"counter-agg-irate-right\"></a>\n\n```SQL ,ignore\nirate_right(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe instantaneous rate of change of the counter at the right (later) side of the time range. Essentially, the [`idelta_right`](#counter-agg-idelta-right) divided by the duration between the first and second observed points in the `CounterSummary`. This can be especially useful for fast moving counters.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `irate_right` | `DOUBLE PRECISION` | The instantaneous rate computed from right (later) side of the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-irate-right-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    irate_right(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n---\n# **Counting functions** <a id=\"counter-agg-api-counting\"></a>\nThe counting functions comprise several accessor functions that calculate the number of times a certain thing occurred while calculating the [`counter_agg`](#counter-agg-point).\n\n---\n## **num_changes()** <a id=\"counter-agg-num-changes\"></a>\n\n```SQL ,ignore\nnum_changes(\n    summary CounterSummary\n) RETURNS BIGINT\n```\n\nThe number of times the value changed within the period over which the `CounterSummary` is calculated. This is determined by evaluating consecutive points, any change counts, including counter resets where the counter is reset to zero, while this would result in the same _adjusted_ counter value for consecutive points, we still treat it as a change.\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `num_changes` | `BIGINT` | The number of times the value changed|\n<br>\n\n### Sample Usage <a id=\"counter-agg-num-changes-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    num_changes(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **num_elements()** <a id=\"counter-agg-num-elements\"></a>\n\n```SQL ,ignore\nnum_elements(\n    summary CounterSummary\n) RETURNS BIGINT\n```\n\nThe total number of points we saw in calculating the `CounterSummary`. Only points with distinct times are counted, as duplicate times are thrown out in general in these calculations.\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input `CounterSummary` from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `num_elements` | `BIGINT` | The number of points seen during the [`counter_agg`](#counter-agg-point) call|\n<br>\n\n### Sample Usage <a id=\"counter-agg-num-elements-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    num_elements(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **num_resets()** <a id=\"counter-agg-num-resets\"></a>\n\n```SQL ,ignore\nnum_resets(\n    summary CounterSummary\n) RETURNS BIGINT\n```\n\nThe total number of times we detected a counter reset while calculating the `CounterSummary`.\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input `CounterSummary` from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `num_resets` | `BIGINT` | The number of resets detected during the [`counter_agg`](#counter-agg-point) call|\n<br>\n\n### Sample Usage <a id=\"counter-agg-num-resets-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    num_resets(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n---\n# **Statistical regression functions** <a id=\"counter-agg-api-regression-fam\"></a>\nThe statistical regression family of functions contains several functions derived from a least squares fit of the adjusted value of the counter. All counter values have resets accounted for before being fed into the linear regression algorithm (and any combined `CounterSummaries` have the proper adjustments performed for resets to enable the proper regression analysis to be performed).\n\n###### NB: Note that the timestamps input are converted from their their internal representation (microseconds since the Postgres Epoch (which is 2000-01-01 00:00:00+00, for some reason), to double precision numbers representing seconds from the Postgres Epoch, with decimal places as fractional seconds, before running the linear regression. Because the internal representation of the timestamp is actually 64-bit integer representing microseconds from the Postgres Epoch, it provides more precision for very large timestamps (the representable range goes out to 294276-12-31). If you want to have accurate, microsecond level precision on your regression analysis dealing with dates at the edge of this range (first off, who are you and *what the heck are you working on???*) we recommend subtracting a large static date from your timestamps and then adding it back after the analysis has concluded. Very small timestamps should be fine as the range does not extend beyond 4714-11-01 BCE, beyond which Julian dates [are not considered reliable by Postgres](https://github.com/postgres/postgres/blob/c30f54ad732ca5c8762bb68bbe0f51de9137dd72/src/include/datatype/timestamp.h#L131). This means that the negative integers are not fully utilized in the timestamp representation and you don't have to worry about imprecision in your computed slopes if you have traveled back in time and are timing chariot races to the microsecond. However, if you travel much further back in time, you're still SOL, as we can't represent the timestamp in the Julian calendar.\n\n---\n## **slope()** <a id=\"counter-agg-slope\"></a>\n\n```SQL ,ignore\nslope(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe slope of the least squares fit line computed from the adjusted counter values and times input in the `CounterSummary`. Because the times are input as seconds, the slope will provide a per-second rate of change estimate based on the least squares fit, which will often be similar to the result of the `rate` calculation, but may more accurately reflect the \"usual\" behavior if there are infrequent, large changes in a counter.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `slope` | `DOUBLE PRECISION` | The per second rate of change computed by taking the slope of the least squares fit of the points input in the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-slope-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    slope(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n---\n## **intercept()** <a id=\"counter-agg-intercept\"></a>\n\n```SQL ,ignore\nintercept(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe intercept of the least squares fit line computed from the adjusted counter values and times input in the `CounterSummary`. This will correspond to the projected value at the Postgres Epoch (2000-01-01 00:00:00+00) - which is not all that useful for much of anything except potentially drawing the best fit line on a graph, using the slope and the intercept.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `intercept` | `DOUBLE PRECISION` | The intercept of the least squares fit line computed from the points input to the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-intercept-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    intercept(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n---\n## **counter_zero_time()** <a id=\"counter-agg-counter-zero-time\"></a>\n\n```SQL ,ignore\ncounter_zero_time(\n    summary CounterSummary\n) RETURNS TIMESTAMPTZ\n```\n\nThe time at which the counter value is predicted to have been zero based on the least squares fit line computed from the points in the `CounterSummary`. The\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `counter_zero_time` | `TIMESTAMPTZ` | The time at which the counter value is predicted to have been zero based on the least squares fit of the points input to the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-counter-zero-time-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    counter_zero_time(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n---\n## **corr())** <a id=\"counter-agg-corr\"></a>\n\n```SQL ,ignore\ncorr(\n    summary CounterSummary\n) RETURNS DOUBLE PRECISION\n```\n\nThe correlation coefficient of the least squares fit line of the adjusted counter value. Given that the slope a line for any counter value must be non-negative, this will also always be non-negative and in the range from [0.0, 1.0] It measures how well the least squares fit fit the available data, where a value of 1.0 represents the strongest correlation between time the counter increasing.\n\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input CounterSummary from a [`counter_agg`](#counter-agg-point) call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `corr` | `DOUBLE PRECISION` | The correlation coefficient computed from the least squares fit of the adjusted counter values input to the `CounterSummary`|\n<br>\n\n### Sample Usage <a id=\"counter-agg-corr-sample\"></a>\n\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    corr(summary)\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n\n# **Utility Functions** <a id=\"counter-agg-api-utilities\"></a>\n---\n## **with_bounds() **<a id=\"counter-agg-with-bounds\"></a>\n```SQL ,ignore\nwith_bounds(\n    summary CounterSummary,\n    bounds TSTZRANGE,\n) RETURNS CounterSummary\n```\n\nA utility function to add bounds to an already-computed `CounterSummary`. The bounds represent the outer limits of the timestamps allowed for this `CounterSummary` as well as the edges of the range to extrapolate to in functions that do that.\n\n### Required Arguments\n|Name| Type |Description|\n|---|---|---|\n| `summary` | `CounterSummary` | The input `CounterSummary`,\n| `bounds` | `TSTZRANGE` |  A range of `timestamptz` representing the largest and smallest allowed times in this `CounterSummary` |\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `counter_agg` | `CounterSummary` |  A CounterSummary object that can be passed to [accessor functions](#counter-agg-api-accessors) or other objects in the counter aggregate API|\n<br>\n\n### Sample Usage\n```SQL ,ignore\nSELECT\n    id,\n    bucket,\n    extrapolated_rate(\n        with_bounds(\n            summary,\n            time_bucket_range('15 min'::interval, bucket)\n        )\n    )\nFROM (\n    SELECT\n        id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        counter_agg(ts, val) AS summary\n    FROM foo\n    GROUP BY id, time_bucket('15 min'::interval, ts)\n) t\n```\n---\n# Notes on Parallelism and Ordering <a id=\"counter-agg-ordering\"></a>\n\nThe counter reset calculations we perform require a strict ordering of inputs and therefore the calculations are not parallelizable in the strict Postgres sense. This is because when Postgres does parallelism it hands out rows randomly, basically as it sees them to workers. However, if your parallelism can guarantee disjoint (in time) sets of rows, the algorithm can be parallelized, just so long as within some time range, all rows go to the same worker. This is the case for both [continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates) and for [distributed hypertables](https://docs.timescale.com/latest/using-timescaledb/distributed-hypertables) (as long as the partitioning keys are in the group by, though the aggregate itself doesn't horribly make sense otherwise).\n\nWe throw an error if there is an attempt to combine overlapping `CounterSummaries`, for instance, in our example above, if you were to try to combine summaries across `measure_id`'s it would error (assuming that they had overlapping times). This is because the counter values resetting really only makes sense within a given time series determined by a single `measure_id`. However, once an accessor function is applied, such as `delta`, a sum of deltas may be computed. Similarly, an average or histogram of rates across multiple time series might be a useful calculation to perform. The thing to note is that the counter aggregate and the reset logic should be performed first, then further calculations may be performed on top of that.\n\nAs an example, let's consider that we might want to find which of my counters had the most extreme rates of change in each 15 minute period. For this, we'll want to normalize the rate of change of each measure by dividing it by the average rate of change over all the counters in that 15 minute period. We'll use the normal `avg` function to do this, but we'll use it as a window function like so:\n\n\n```SQL ,ignore\nWITH t as (SELECT measure_id,\n        time_bucket('15 min'::interval, ts) AS bucket,\n        rate(\n            counter_agg(ts, val)\n        ) as rate\n    FROM foo\n    GROUP BY measure_id),\nSELECT measure_id,\n    bucket,\n    rate,\n    rate / avg(rate_per_measure) OVER (PARTITION BY bucket) AS normalized_rate -- call normal avg function as a window function to get a 15 min avg to normalize our per-measure rates\nFROM t;\n```\nStill, note that the counter resets are accounted for before applying the `avg` function in order to get our normalized rate.\n\nInternally, the `CounterSummary` stores:\n- the first, second, penultimate, and last points seen\n- the sum of all the values at reset points, as well as the number of changes, and number of resets seen.\n- A set of 6 values used to compute all the statistical regression parameters using the Youngs-Cramer algorithm.\n- Optionally, the bounds as an open-ended range, over which extrapolation should occur and which represents the outer possible limit of times represented in this `CounterSummary`\n\nIn general, the functions support [partial aggregation](https://www.postgresql.org/docs/current/xaggr.html#XAGGR-PARTIAL-AGGREGATES) and partitionwise aggregation in the multinode context, but are not parallelizable (in the Postgres sense, which requires them to accept potentially overlapping input).\n\nBecause they require ordered sets, the aggregates build up a buffer of input data, sort it and then perform the proper aggregation steps. In cases where memory is proving to be too small to build up a buffer of points causing OOMs or other issues, a multi-level aggregate can be useful.\n\nSo where I might run into OOM issues if I computed the values over all time like so:\n\n\n```SQL ,ignore\nSELECT measure_id,\n    rate(\n        counter_agg(ts, val)\n    ) as rate\nFROM foo\nGROUP BY measure_id;\n```\nIf I were to instead, compute the [`counter_agg`](#counter-agg-point) over, say daily buckets and then combine the aggregates, I might be able to avoid OOM issues, as each day will be computed separately first and then combined, like so:\n\n```SQL ,ignore\nWITH t as (SELECT measure_id,\n        time_bucket('1 day'::interval, ts) AS bucket,\n        counter_agg(ts, val)\n    FROM foo\n    GROUP BY measure_id),\nSELECT measure_id, \\\n    rate(\n        rollup(counter_agg) --combine the daily `CounterSummaries` to make a full one over all time, accounting for all the resets, then apply the rate function\n    )\nFROM t;\n```\n\nMoving aggregate mode is not supported by [`counter_agg`](#counter-agg-point) and its use as a window function may be quite inefficient.\n\n---\n# Extrapolation Methods Details <a id=\"counter-agg-methods\"></a>\n#TODO\n"
  },
  {
    "path": "docs/examples/tdigest.c",
    "content": "// cc -o tdigest tdigest.c $CARGO_TARGET_DIR/$PROFILE/libtimescaledb_toolkit_tdigest.a -lm -lpthread -ldl\n\n// Sample program which prints the expected output of the test_tdigest_io test.\n\n////////////////////////////////////////////////////////////////////////////////\n// TODO Generate a header from tdigest-lib crate.\n\n#include <sys/types.h>\n\nstruct TDigestBuilder;\nstruct TDigest;\n\n// Return pointer to new TDigestBuilder.\n// MUST NOT be passed to free(3).  Instead, pass to timescaledb_toolkit_tdigest_builder_free to\n// discard or to timescaledb_toolkit_tdigest_build to convert to TDigest.\n// Never returns NULL.\nstruct TDigestBuilder *\ntimescaledb_toolkit_tdigest_builder_with_size(size_t size);\n\nvoid\ntimescaledb_toolkit_tdigest_push(struct TDigestBuilder *builder, double value);\n\nvoid\ntimescaledb_toolkit_tdigest_merge(struct TDigestBuilder *builder, struct TDigestBuilder *other);\n\n// Free a TDigestBuilder that has not been built.\n// MUST NOT be passed NULL.\nvoid\ntimescaledb_toolkit_tdigest_builder_free(struct TDigestBuilder *builder);\n\n// Return pointer to new TDigest built from builder.\n// builder MUST NOT be passed to timescaledb_toolkit_tdigest_builder_free .\nstruct TDigest *\ntimescaledb_toolkit_tdigest_build(struct TDigestBuilder *builder);\n\n// Free a TDigest.\nvoid\ntimescaledb_toolkit_tdigest_free(struct TDigest *td);\n\n// Return pointer to null-terminated buffer containing ASCII serialization of TDigest suitable for\n// use with postgresql INSERT.\n// Free the buffer with free(3).\nchar *\ntimescaledb_toolkit_tdigest_format_for_postgres(struct TDigest *td);\n\n////////////////////////////////////////////////////////////////////////////////\n\n#include <stdio.h>\n#include <stdlib.h>\n\nint\nmain()\n{\n    struct TDigestBuilder *builder = timescaledb_toolkit_tdigest_builder_with_size(100);\n    double value;\n    for (value = 1.0; value <= 100.0; value++) {\n        timescaledb_toolkit_tdigest_push(builder, value);\n    }\n\n    struct TDigest *td = timescaledb_toolkit_tdigest_build(builder);\n    char *formatted = timescaledb_toolkit_tdigest_format_for_postgres(td);\n    printf(\"%s\\n\", formatted);\n    free(formatted);\n\n    timescaledb_toolkit_tdigest_free(td);\n\n    return 0;\n}\n"
  },
  {
    "path": "docs/examples/tdigest.py",
    "content": "import ctypes\nimport os\n\n_cdll = ctypes.CDLL(os.path.join(\n    os.getenv('CARGO_TARGET_DIR', 'target'),\n    os.getenv('PROFILE', 'debug'),\n    'libtimescaledb_toolkit_tdigest.so'))\n_cdll.timescaledb_toolkit_tdigest_builder_with_size.restype = ctypes.c_void_p\n_cdll.timescaledb_toolkit_tdigest_build.restype =  ctypes.c_void_p\n_cdll.timescaledb_toolkit_tdigest_format_for_postgres.restype = ctypes.POINTER(ctypes.c_char)\n_cdll.timescaledb_toolkit_tdigest_push.restype = None\n_cdll.timescaledb_toolkit_tdigest_merge.restype = None\n_cdll.timescaledb_toolkit_tdigest_builder_free.restype = None\n_cdll.timescaledb_toolkit_tdigest_free.restype = None\n\n# Wrapper classes use `real_pointer` to keep hold of the real pointer for as\n# long as it needs to be released.\n# We copy it to self.pointer to enforce use of `with` (as much as anything can be enforced in Python).\n# Attempting to forego `with` results in `AttributeError`.\n\nclass TDigest:\n\n    class Builder:\n        def __init__(self, pointer):\n            self.real_pointer = pointer\n\n        def __enter__(self):\n            self.pointer = self.real_pointer\n            return self\n\n        def __exit__(self, exc_type, exc_val, exc_tb):\n            self.__del__()\n            self.real_pointer = None\n            if 'pointer' in self.__dict__:\n                del self.__dict__['pointer']\n\n        def __del__(self):\n            if self.real_pointer is not None:\n                _cdll.timescaledb_toolkit_tdigest_builder_free(self.real_pointer)\n\n        def with_size(size):\n            return TDigest.Builder(ctypes.c_void_p(_cdll.timescaledb_toolkit_tdigest_builder_with_size(ctypes.c_size_t(size))))\n\n        def push(self, value):\n            _cdll.timescaledb_toolkit_tdigest_push(self.pointer, ctypes.c_double(value))\n\n        def build(self):\n            td = TDigest(ctypes.c_void_p(_cdll.timescaledb_toolkit_tdigest_build(self.pointer)))\n            self.real_pointer = None\n            del self.__dict__['pointer']\n            return td\n\n    def __init__(self, pointer):\n        self.real_pointer = pointer\n\n    def __enter__(self):\n        self.pointer = self.real_pointer\n        return self\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        self.__del__()\n        self.real_pointer = None\n        if 'pointer' in self.__dict__:\n            del self.__dict__['pointer']\n\n    def __del__(self):\n        if self.real_pointer is not None:\n            _cdll.timescaledb_toolkit_tdigest_free(self.real_pointer)\n\n    def format_for_postgres(self):\n        buf = _cdll.timescaledb_toolkit_tdigest_format_for_postgres(self.pointer)\n        s = ctypes.cast(buf, ctypes.c_char_p).value.decode('ascii')\n        # TODO free(3) left as an exercise to the reader.  This is for GNU libc on Linux/amd64:\n        ctypes.CDLL('libc.so.6').free(buf)\n        return s\n\n# Sample program which prints the expected output of the test_tdigest_io test.\ndef test():\n    with TDigest.Builder.with_size(100) as builder:\n        for value in range(1, 101):\n            builder.push(value)\n        with builder.build() as td:\n            print(td.format_for_postgres())\n\nif __name__ == '__main__':\n    test()\n"
  },
  {
    "path": "docs/gauge_agg.md",
    "content": "# Gauge Aggregates [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes)\n\nA gauge is a metric similar to a counter, with the primary difference being\nthat it measures a value that varies up and down over time, rather than an\never-increasing COUNT of the number of times something happened.\nExamples include resource utilization metrics, precipitation levels,\nor temperatures.\n\n`gauge_agg` currently shares implementation with `counter_agg` but without the\nresetting logic.  This means it enforces ordering even though that is not\nnecessarily required for all gauge aggregates.  We may offer an additional\nunordered gauge aggregate in the future.\n\n# Test table\n\nExamples below are tested against the following table:\n\n```SQL ,non-transactional\nSET TIME ZONE 'UTC';\nCREATE TABLE gauge_test (\n    measure_id      BIGINT,\n    ts              TIMESTAMPTZ ,\n    val             DOUBLE PRECISION,\n    PRIMARY KEY (measure_id, ts)\n);\nINSERT INTO gauge_test SELECT 1, '2020-01-03 UTC'::timestamptz + make_interval(days=>v), v + 1000 FROM generate_series(1,10) v;\nINSERT INTO gauge_test SELECT 2, '2020-01-03 UTC'::timestamptz + make_interval(days=>v), v + 2000 FROM generate_series(1,10) v;\nINSERT INTO gauge_test SELECT 3, '2020-01-03 UTC'::timestamptz + make_interval(days=>v), v + 3000 FROM generate_series(1,10) v;\n```\n\n## Functions\n\n### delta\n\n```SQL,ignore\nSELECT toolkit_experimental.delta(toolkit_experimental.gauge_agg(ts, val)) FROM gauge_test;\n```\n```ignore\n delta\n-------\n -1991\n```\n\n### idelta_left\n\n```SQL,ignore\nSELECT toolkit_experimental.idelta_left(toolkit_experimental.gauge_agg(ts, val)) FROM gauge_test;\n```\n```ignore\n idelta_left\n-------------\n        1002\n```\n\n### idelta_right\n\n```SQL,ignore\nSELECT toolkit_experimental.idelta_right(toolkit_experimental.gauge_agg(ts, val)) FROM gauge_test;\n```\n```ignore\n idelta_right\n--------------\n         1010\n```\n\n### rollup\n\n```SQL\nWITH t as (SELECT date_trunc('minute', ts), toolkit_experimental.gauge_agg(ts, val) as agg FROM gauge_test group by 1)\n    SELECT toolkit_experimental.delta(toolkit_experimental.rollup(agg)) FROM t;\n```\n```output\n rollup delta\n--------------\n            9\n```\n"
  },
  {
    "path": "docs/hyperloglog.md",
    "content": "# Hyperloglog\n\n> [Description](#hyperloglog-description)<br>\n> [Details](#hyperloglog-details)<br>\n> [API](#hyperloglog-api)\n\n## Description <a id=\"hyperloglog-description\"></a>\n\nTimescaleDB Toolkit provides an implementation of the [Hyperloglog estimator](https://en.wikipedia.org/wiki/HyperLogLog) for `COUNT DISTINCT` approximations of any type that has a hash function.\n\n## Details <a id=\"hyperloglog-details\"></a>\n\nTimescale's HyperLogLog is implemented as an aggregate function in PostgreSQL.  They do not support moving-aggregate mode, and are not ordered-set aggregates.  It is restricted to values that have an extended hash function.  They are partializable and are good candidates for [continuous aggregation](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).\n\n\n## Command List (A-Z) <a id=\"hyperloglog-api\"></a>\n> - [hyperloglog](#hyperloglog)\n> - [distinct_count](#distinct_count)\n\n---\n## **hyperloglog** <a id=\"hyperloglog\"></a>\n```SQL,ignore\nhyperloglog(\n    size INTEGER,\n    value AnyElement¹\n) RETURNS Hyperloglog\n```\n¹The type must have an extended (64bit) hash function.\n\nThis will construct and return a Hyperloglog with at least the specified number of buckets over the given values.\n\n### Required Arguments <a id=\"hyperloglog-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `buckets` | `INTEGER` | Number of buckets in the digest. Will be rounded up to the next power of 2, must be between 16 and 2^18. Increasing this will usually provide more accurate at the expense of more storage. |\n| `value` | `AnyElement` |  Column to count the distinct elements of. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `hyperloglog` | `Hyperloglog` | A hyperloglog object which may be passed to other hyperloglog APIs. |\n<br>\n\n### Sample Usages <a id=\"hyperloglog-examples\"></a>\nFor this examples assume we have a table 'samples' with a column 'weights' holding `DOUBLE PRECISION` values.  The following will simply return a digest over that column\n\n```SQL ,ignore\nSELECT hyperloglog(64, weights) FROM samples;\n```\n\nIt may be more useful to build a view from the aggregate that we can later pass to other tdigest functions.\n\n```SQL ,ignore\nCREATE VIEW digest AS SELECT hyperloglog(64, data) FROM samples;\n```\n\n---\n\n## **rollup** <a id=\"rollup\"></a>\n\n```SQL ,ignore\nrollup(\n    log hyperloglog\n) RETURNS Hyperloglog\n```\n\nReturns a Hyperloglog by aggregating over the union of the input elements.\n\n### Required Arguments <a id=\"hyperloglog-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `log` | `Hyperloglog` |  Column of Hyperloglogs to be unioned. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `hyperloglog` | `Hyperloglog` | A hyperloglog containing the count of the union of the input Hyperloglogs. |\n<br>\n\n### Sample Usages <a id=\"summary-form-examples\"></a>\n\n```SQL\nSELECT distinct_count(rollup(logs))\nFROM (\n    (SELECT hyperloglog(32, v::text) logs FROM generate_series(1, 100) v)\n    UNION ALL\n    (SELECT hyperloglog(32, v::text) FROM generate_series(50, 150) v)\n) hll;\n```\n```output\n count\n-------\n   153\n```\n\n---\n\n## **distinct_count** <a id=\"distinct_count\"></a>\n```SQL ,ignore\ndistinct_count(hyperloglog Hyperloglog) RETURNS BIGINT\n```\n\nGet the number of distinct values from a hyperloglog.\n\n### Required Arguments <a id=\"distinct_count-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `hyperloglog` | `Hyperloglog` | The hyperloglog to extract the count from. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `distinct_count` | `BIGINT` | The number of distinct elements counted by the hyperloglog. |\n<br>\n\n### Sample Usages <a id=\"distinct_count-examples\"></a>\n\n```SQL\nSELECT distinct_count(hyperloglog(64, data))\nFROM generate_series(1, 100) data\n```\n```output\n distinct_count\n----------------\n            104\n```\n\n## **stderror** <a id=\"hyperloglog_stderror\"></a>\n\n```SQL ,ignore\nstderror(hyperloglog Hyperloglog) RETURNS DOUBLE PRECISION\n```\n\nReturns an estimate of the relative stderror of the hyperloglog based on the\nhyperloglog error formula. Approximate result are:\n```\n precision ┃ registers ┃  error ┃  bytes\n━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━╋━━━━━━━━\n         4 ┃        16 ┃ 0.2600 ┃     12\n         5 ┃        32 ┃ 0.1838 ┃     24\n         6 ┃        64 ┃ 0.1300 ┃     48\n         7 ┃       128 ┃ 0.0919 ┃     96\n         8 ┃       256 ┃ 0.0650 ┃    192\n         9 ┃       512 ┃ 0.0460 ┃    384\n        10 ┃      1024 ┃ 0.0325 ┃    768\n        11 ┃      2048 ┃ 0.0230 ┃   1536\n        12 ┃      4096 ┃ 0.0163 ┃   3072\n        13 ┃      8192 ┃ 0.0115 ┃   6144\n        14 ┃     16384 ┃ 0.0081 ┃  12288\n        15 ┃     32768 ┃ 0.0057 ┃  24576\n        16 ┃     65536 ┃ 0.0041 ┃  49152\n        17 ┃    131072 ┃ 0.0029 ┃  98304\n        18 ┃    262144 ┃ 0.0020 ┃ 196608\n```\n\n### Required Arguments <a id=\"hyperloglog_stderror-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `hyperloglog` | `Hyperloglog` | The hyperloglog to extract the count from. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `stderror` | `BIGINT` | The number of distinct elements counted by the hyperloglog. |\n<br>\n\n### Sample Usages <a id=\"hyperloglog_stderror-examples\"></a>\n\n```SQL\nSELECT stderror(hyperloglog(64, data))\nFROM generate_series(1, 100) data\n```\n```output\n stderror\n----------\n     0.13\n```\n"
  },
  {
    "path": "docs/lttb.md",
    "content": "# Largest Triangle Three Buckets\n\n> [Description](#description)<br>\n> [Example](#example)<br>\n> [API](#api)\n\n## Description <a id=\"description\"></a>\n\n[Largest Triangle Three Buckets](https://github.com/sveinn-steinarsson/flot-downsample)\nis a downsampling method that tries to retain visual similarity between the\ndownsampled data and the original dataset. TimescaleDB Toolkit provides an\nimplementation of this which takes `(timestamp, value)` pairs, sorts them if\nneeded, and downsamples them.\n\n\n## Usage Example <a id=\"details\"></a>\n\nIn this example we're going to examine downsampling a 101 point cosine wave\ngenerated like so\n\n```SQL ,non-transactional\nSET TIME ZONE 'UTC';\nCREATE TABLE sample_data(time TIMESTAMPTZ, val DOUBLE PRECISION);\nINSERT INTO sample_data\n    SELECT\n        '2020-01-01 UTC'::TIMESTAMPTZ + make_interval(days=>(foo*10)::int) as time,\n        10 + 5 * cos(foo) as val\n    FROM generate_series(1,11,0.1) foo\n```\n```output\nINSERT 0 101\n```\n\nwhen graphed, this waves appears like so\n\n![Raw data](images/lttb_raw.png)\n\nwe can downsample it to various degrees using `lttb`, for instance, downsampling\nto 34 points\n\n```SQL\nSELECT time, value::numeric(10,2)\nFROM unnest((\n    SELECT lttb(time, val, 34)\n    FROM sample_data))\n```\n\n<div hidden>\n\n```output\n          time          | value\n------------------------+-------\n 2020-01-11 00:00:00+00 | 12.70\n 2020-01-13 00:00:00+00 | 11.81\n 2020-01-15 00:00:00+00 | 10.85\n 2020-01-19 00:00:00+00 |  8.86\n 2020-01-22 00:00:00+00 |  7.48\n 2020-01-25 00:00:00+00 |  6.31\n 2020-01-28 00:00:00+00 |  5.48\n 2020-01-31 00:00:00+00 |  5.05\n 2020-02-03 00:00:00+00 |  5.06\n 2020-02-06 00:00:00+00 |  5.52\n 2020-02-09 00:00:00+00 |  6.37\n 2020-02-12 00:00:00+00 |  7.55\n 2020-02-15 00:00:00+00 |  8.95\n 2020-02-20 00:00:00+00 | 11.42\n 2020-02-23 00:00:00+00 | 12.77\n 2020-02-26 00:00:00+00 | 13.88\n 2020-02-29 00:00:00+00 | 14.64\n 2020-03-03 00:00:00+00 | 14.98\n 2020-03-06 00:00:00+00 | 14.88\n 2020-03-09 00:00:00+00 | 14.35\n 2020-03-11 00:00:00+00 | 13.77\n 2020-03-14 00:00:00+00 | 12.63\n 2020-03-17 00:00:00+00 | 11.26\n 2020-03-22 00:00:00+00 |  8.78\n 2020-03-25 00:00:00+00 |  7.40\n 2020-03-28 00:00:00+00 |  6.26\n 2020-03-31 00:00:00+00 |  5.44\n 2020-04-03 00:00:00+00 |  5.04\n 2020-04-06 00:00:00+00 |  5.08\n 2020-04-09 00:00:00+00 |  5.55\n 2020-04-11 00:00:00+00 |  6.10\n 2020-04-14 00:00:00+00 |  7.20\n 2020-04-17 00:00:00+00 |  8.54\n 2020-04-20 00:00:00+00 | 10.02\n```\n\n</div>\n\n```\noutput omitted\n```\n\nlooks like so\n\n![Raw data](images/lttb_34.png)\n\nas you further downsample, you retain fewer and fewer datapoints, and the\nresulting data looks less and less like the original\n\n```SQL\nSELECT time, value::numeric(10,2)\nFROM unnest((\n    SELECT lttb(time, val, 17)\n    FROM sample_data))\n```\n```output\n          time          | value\n------------------------+-------\n 2020-01-11 00:00:00+00 | 12.70\n 2020-01-13 00:00:00+00 | 11.81\n 2020-01-22 00:00:00+00 |  7.48\n 2020-01-28 00:00:00+00 |  5.48\n 2020-02-03 00:00:00+00 |  5.06\n 2020-02-09 00:00:00+00 |  6.37\n 2020-02-14 00:00:00+00 |  8.46\n 2020-02-24 00:00:00+00 | 13.17\n 2020-03-01 00:00:00+00 | 14.80\n 2020-03-07 00:00:00+00 | 14.75\n 2020-03-13 00:00:00+00 | 13.04\n 2020-03-23 00:00:00+00 |  8.30\n 2020-03-29 00:00:00+00 |  5.94\n 2020-04-04 00:00:00+00 |  5.00\n 2020-04-10 00:00:00+00 |  5.80\n 2020-04-14 00:00:00+00 |  7.20\n 2020-04-20 00:00:00+00 | 10.02\n```\n\n![Raw data](images/lttb_17.png)\n\n\n```SQL\nSELECT time, value::numeric(10,2)\nFROM unnest((\n    SELECT lttb(time, val, 8)\n    FROM sample_data))\n```\n```output\n          time          | value\n------------------------+-------\n 2020-01-11 00:00:00+00 | 12.70\n 2020-01-27 00:00:00+00 |  5.72\n 2020-02-06 00:00:00+00 |  5.52\n 2020-02-27 00:00:00+00 | 14.17\n 2020-03-09 00:00:00+00 | 14.35\n 2020-03-30 00:00:00+00 |  5.67\n 2020-04-09 00:00:00+00 |  5.55\n 2020-04-20 00:00:00+00 | 10.02\n```\n\n![Raw data](images/lttb_8.png)\n\n## Command List (A-Z) <a id=\"api\"></a>\n> - [lttb](#lttb)\n\n---\n## **lttb** <a id=\"lttb\"></a>\n```SQL,ignore\nlttb(\n    time TIMESTAMPTZ,\n    value DOUBLE PRECISION,\n    resolution INTEGER\n) RETURNS SortedTimevector\n```\n\nThis will construct and return a sorted timevector with at most `resolution`\npoints. `unnest(...)` can be used to\nextract the `(time, value)` pairs from this series\n\n### Required Arguments <a id=\"lttb-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `time` | `TIMESTAMPTZ` | Time (x) value for the data point. |\n| `value` | `DOUBLE PRECISION` |  Data (y) value for the data point. |\n| `resolution` | `INTEGER` | Number of points the output should have. |\n<br>\n\n### Sample Usage <a id=\"lttb-examples\"></a>\n\n```SQL\nSELECT time, value\nFROM unnest((\n    SELECT lttb(time, val, 4)\n    FROM sample_data))\n```\n```output\n          time          |       value\n------------------------+--------------------\n 2020-01-11 00:00:00+00 |   12.7015115293407\n 2020-02-01 00:00:00+00 |  5.004324248633603\n 2020-03-03 00:00:00+00 | 14.982710485116087\n 2020-04-20 00:00:00+00 | 10.022128489940254\n```"
  },
  {
    "path": "docs/ordered-aggregates.md",
    "content": "# Implementing aggregates that require ordered inputs\n\nPostgreSQL has a couple different ways of dealing with aggregates that require ordered inputs, [ordered set aggregates](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE), which guarantee ordered input but have non-intuitive syntax. You can also specify an ordering within an aggregate call (ie `SELECT array_agg(foo ORDER BY foo, bar)`), however, AFAIK the aggregate doesn't know and has no way of enforcing that this ordering has occurred other than balking if it got out of order data. \n\nBoth of these have rather annoying syntax and require the *user* to understand that the input needs to be ordered for the aggregate to function. We decided that this was a poor choice. Instead, we decided to do the ordering ourselves *inside* the aggregate function. This means that the transition function for any of the aggregates that require ordering to function (`time_weight`, `counter_agg` etc) first have a transition function that simply builds up an array of inputs to the aggregate, then sorts the array and then processes the inputs in order. \n\nIn addition, these aggregates have different semantics for combine and rollup than some of our other functions. Once the data has been sorted and processed, in general, these aggregates can *only* be combined in the traditional sense if they contain disjoint regions of time, in other words, only aggregates covering non-overlapping periods of time can be rolled up or combined. \n\nPostgreSQL doesn't have a way to guarantee that only non-overlapping time periods can be sent to each parallel worker, rows are distributed essentially as they are seen in a round robin. This means that the aggregates cannot be marked as parallel safe. So then, why do they need combine functions at all? Well, there is another time when combine functions are called and that is in the case of partitionwise aggregation. Partitionwise aggregation is used to perform part of the aggregation on a particular partition and then take the state and combine with aggregates from other partitions. Partitions are disjoint in time for us (this assumes some things and we should still have checks to make sure that we are not getting out of order / overlapping data). We believe the test for this is whether they have a combine function, not whether they are marked parallel safe. Therefore, we always mark these aggregates as parallel restricted rather than parallel safe, which hopefully will allow them to be used for partitionwise but not parallel aggregates. Partitionwise aggregation is a potential large optimization area for multinode so we wanted to make sure we could support that case. \n\nThis also impacts the way that `rollup` can be called on these functions and the cases in which we should error. \n\nNote also that the `combine` and `rollup` functions for these aggregates must do essentially the same thing that the transition function does and build up an array of partial states, then order them and combine them at the end. This is a bit odd, but seems to be the best way. \n\n## Implementation example\n\nHere is the rollup aggregate for `TimeWeightSummary`:\n```SQL , ignore\nCREATE AGGREGATE rollup(tws TimeWeightSummary)\n(\n    sfunc = time_weight_summary_trans,\n    stype = internal,\n    finalfunc = time_weight_final,\n    combinefunc = time_weight_combine,\n    serialfunc = time_weight_trans_serialize,\n    deserialfunc = time_weight_trans_deserialize,\n    parallel = restricted\n);\n```\n\n### Parallel safety\nThe aggregate above is marked as `parallel = restricted`, which specifies that [\"the function can be executed in parallel mode, but the execution is restricted to parallel group leader\"](https://www.postgresql.org/docs/current/sql-createfunction.html). Note that only the value of the `parallel` parameter of the `CREATE AGGREGATE` call is used for determining the parallel safety of the aggregate; the parallel safetyness of the support functions that make up the aggregate are ignored when the aggregate is called. But all support functions should be marked parallel safe because, AFAIK, they are immutable and parallel safe in all cases, it is only when they are called in the correct ordering with the aggregate that they can cause problems / error if not used correctly. \n\n### Merging on serialization\n\nIn many cases the implementation of aggregate merging requires that the aggregates to be merged cover non-overlapping periods of time. To handle this while allowing the inputs to be potentially unordered, in the aggregate:\n- the transition function appends the input to a `Vec`\n- the final function sorts the transition state and merges all of the elements\n\nStoring all of the inputs ever seen in the transition state takes up a lot of memory, and makes the final function use a lot of compute. We can partially alleviate those issues by:\n\n- Adding a `combinefunc` that appends the second transition state `Vec` to the first one\n- Adding a `serialfunc` that:\n  1. Sorts and merges the transition state\n  2. Serializes the transition state\n - Adding a `deserialfunc` that deserializes the transition state\n\nThese extra functions improve performance when the inputs are partitioned since each partition is combined, and then the partition combinations are combined again.\n\n`serialfunc` is called right before sending the current transition state from the parallel worker to the parent process, so it's the only place where we can do the sorting/merging of the transition state before it gets sent to the parent process. We do the merging in the parallel worker to reduce the amount of data sent from the parallel worker to the parent process.\n\n![Each group of days is sorted and merged, then each group is sorted and merged](images/pgmerging.svg)\n\nThis method doesn't work when two partitions contain overlapping time ranges. That shouldn't happen when the partitions are chunks of a TimescaleDB hypertable, but it could happen when the partitions cover overlapping segments of time (e.g. a table that uses declarative partitioning to partition a table using the hash of an ID). When two partitions contain overlapping time ranges, the implementation should catch that and give an error.\n\nNote that this approach means that `deserialfunc(serialfunc(x)) != x`, which is weird but doesn't seem to cause any problems.\n\n\n"
  },
  {
    "path": "docs/percentile_approximation.md",
    "content": "# Approximate Percentiles\n> [Why To Use Approximate Percentiles](#why-use)<br>\n> [API](#percentile-approx-api) <br>\n> [Advanced Usage: Algorithms and How to Choose](#advanced-usage)<br>\n\n###### A note on terminology: Technically, a percentile divides the group into 100 equally sized (by frequency) buckets, while a quantile would divide the group into an arbitrary number of buckets. We use percentile here with the recognition that while quantile is the technically more \"correct\" term for an arbitrary precision operation, percentile has become more commonly used to describe this type of function.\n\n## Why to Use Approximate Percentiles <a id=\"why-use\"></a>\n\nThere are really two things to cover here:  1) [why use percentiles at all](#why-use-percent) and 2) [why use *approximate* percentiles rather than exact percentiles](#why-approximate).\n\nTo better understand this, we'll use the common example of a server that's running APIs for a company and tracking the response times for the various APIs it's running. So, for our example, we have a table something like this:\n\n```SQL , non-transactional, ignore-output\nSET extra_float_digits = -3; -- use 12 digits of precision to reduce flakiness\nSET SESSION TIME ZONE 'UTC'; -- so we get consistent output\nCREATE TABLE response_times (\n    ts timestamptz,\n    api_id int,\n    user_id int,\n    response_time_ms float\n);\n-- and we'll make it a hypertable for ease of use in the rest of the example\nSELECT create_hypertable('response_times', 'ts');\n```\n<details> <a id=\"data-generation\"></a>\n    <summary> We'll also generate some data to work with here. And insert it into the table (expand for the generation script if you want to see it). </summary>\n\n```SQL , non-transactional, ignore-output\nWITH apis as MATERIALIZED (SELECT generate_series(1, 12) as api_id),\nusers as MATERIALIZED (SELECT generate_series(1, 30) as user_id),\napi_users as MATERIALIZED (SELECT * FROM apis JOIN users on api_id % 3 = user_id % 3),  -- users use ~ 1/3 of apis\ntimes as MATERIALIZED (SELECT generate_series('2020-01-01'::timestamptz, '2020-01-02'::timestamptz, '1 minute'::interval) as ts),\nraw_joined as MATERIALIZED (SELECT * from api_users CROSS JOIN times ORDER BY api_id, user_id, ts),\ngenerated_data as MATERIALIZED (\nSELECT ts + '5 min'::interval * test_random() as ts,\n    api_id,\n    user_id,\n    10 * api_id * user_id   / (1+(extract(hour FROM ts)::int % api_id)) * test_random() as response_time\nFROM raw_joined\nORDER BY api_id, user_id, ts)\n\nINSERT INTO response_times SELECT * FROM generated_data;\n```\nIt's not the most representative of data sets, but it'll do and have some interesting features for us to look at.\n</details>\n\n---\n### Why use percentiles? <a id=\"why-use-percent\"></a>\n\nIn general, percentiles are useful for understanding the distribution of your data, for instance the 50% percentile, aka median of the data can be a more useful measure than average when there are outliers that would dramatically impact the average, but have a much smaller impact on the median. The median or 50th percentile means that in an ordered list of your data half of the data will be greater and half less, the 10% percentile would mean that 10% would fall below and 90% above the value returned and the 99th percentile would mean that 1% is above the value returned, 99% below. Outliers have less of an impact because their magnitude doesn't affect their percentile, only their order in the set, so the skew introduced by uncommon very large or very small values is reduced or eliminated.\n\nLet's look at an example with our generated data set, and lets say we want to find the worst apis, in an hour segment, so that we can identify poor performance, we'll start by using the Postgres [percentile_disc]() function for our percentiles:\n\n```SQL\nSELECT\n    time_bucket('1 h'::interval, ts) as bucket,\n    api_id,\n    avg(response_time_ms),\n    percentile_disc(0.5) WITHIN GROUP (ORDER BY response_time_ms) as median\nFROM response_times\nGROUP BY 1, 2\nORDER BY 3 DESC LIMIT 15;\n```\n```output, precision(2: 7)\n         bucket        | api_id |      avg      |    median\n-----------------------+--------+---------------+--------------\n2020-01-01 00:00:00+00 | 12     | 993.878689655 | 751.68\n2020-01-01 12:00:00+00 | 12     |      948.4199 |  714.6\n2020-01-01 00:00:00+00 | 11     | 848.218549223 |    638\n2020-01-01 22:00:00+00 | 11     | 824.517045075 | 606.32\n2020-01-01 11:00:00+00 | 11     | 824.277392027 | 603.79\n2020-01-01 00:00:00+00 |  9     | 739.073793103 | 562.95\n2020-01-01 00:00:00+00 | 10     | 731.558894646 |  547.5\n2020-01-01 18:00:00+00 |  9     | 724.052854758 | 536.22\n2020-01-01 09:00:00+00 |  9     | 719.944816054 | 529.74\n2020-01-01 20:00:00+00 | 10     | 696.328870432 |  500.8\n2020-01-01 10:00:00+00 | 10     | 694.303472454 |  507.5\n2020-01-01 00:00:00+00 |  8     | 622.262145329 | 466.56\n2020-01-01 08:00:00+00 |  8     | 597.849434276 | 437.12\n2020-01-01 16:00:00+00 |  8     | 597.591488294 | 433.92\n2020-01-02 00:00:00+00 | 11     | 583.857241379 | 383.35\n ```\n\nSo, this returns some interesting results, maybe something like what those of you who read over our [data generation](#data-generation) code would expect. Given how we generate the data, we expect that the larger `api_ids` will have longer generated response times but that it will be cyclic with `hour % api_id`, so we can see that here.\n\nBut what happens if we introduce some aberrant data points? They could have come from anywhere, maybe a user ran a weird query, maybe there's an odd bug in the code that causes some timings to get multiplied in an odd code path, who knows, here we'll introduce just 10 outlier points out of half a million:\n\n```SQL , non-transactional, ignore-output\nWITH rand_points as (SELECT ts, api_id, user_id FROM response_times ORDER BY test_random() LIMIT 10)\nUPDATE response_times SET response_time_ms = 10000 * response_time_ms WHERE (ts, api_id, user_id) IN (SELECT * FROM rand_points);\n```\n```SQL\nSELECT\n    time_bucket('1 h'::interval, ts) as bucket,\n    api_id,\n    avg(response_time_ms),\n    percentile_disc(0.5) WITHIN GROUP (ORDER BY response_time_ms) as median\nFROM response_times\nGROUP BY 1, 2\nORDER BY 3 DESC LIMIT 15;\n```\n\n```output, precision(2: 7)\n         bucket         | api_id |      avg      |     median\n------------------------+--------+---------------+---------------\n2020-01-01 14:00:00+00  |  1     | 1658.34585977 |  53.46\n2020-01-01 06:00:00+00  |  1     | 1226.37258765 |  53.77\n2020-01-01 23:00:00+00  |  1     |     1224.1063 |  53.55\n2020-01-01 00:00:00+00  | 12     | 993.878689655 | 751.68\n2020-01-01 11:00:00+00  |  1     | 961.352933333 |  53.76\n2020-01-01 12:00:00+00  | 12     |      948.4199 |  714.6\n2020-01-01 00:00:00+00  | 11     | 848.218549223 |    638\n2020-01-01 21:00:00+00  |  1     | 846.309280936 |  52.92\n2020-01-01 04:00:00+00  |  1     | 845.378981636 |  54.78\n2020-01-01 22:00:00+00  | 11     | 824.517045075 | 606.32\n2020-01-01 11:00:00+00  | 11     | 824.277392027 | 603.79\n2020-01-01 00:00:00+00  |  9     | 739.073793103 | 562.95\n2020-01-01 00:00:00+00  | 10     | 731.558894646 |  547.5\n2020-01-01 18:00:00+00  |  9     | 724.052854758 | 536.22\n2020-01-01 09:00:00+00  |  9     | 719.944816054 | 529.74\n ```\n\nNow, `avg` is giving horribly misleading results and not showing us the underlying patterns in our data anymore. But if I order by the `median` instead:\n```SQL\nSELECT\n    time_bucket('1 h'::interval, ts) as bucket,\n    api_id,\n    avg(response_time_ms),\n    percentile_disc(0.5) WITHIN GROUP (ORDER BY response_time_ms) as median\nFROM response_times\nGROUP BY 1, 2\nORDER BY 4 DESC, 2, 1 LIMIT 15;\n```\n```output, precision(2: 7)\n         bucket         | api_id |      avg      |    median\n------------------------+--------+---------------+---------------\n2020-01-01 00:00:00+00  | 12     | 993.878689655 | 751.68\n2020-01-01 12:00:00+00  | 12     |      948.4199 |  714.6\n2020-01-01 00:00:00+00  | 11     | 848.218549223 |    638\n2020-01-01 22:00:00+00  | 11     | 824.517045075 | 606.32\n2020-01-01 11:00:00+00  | 11     | 824.277392027 | 603.79\n2020-01-01 00:00:00+00  |  9     | 739.073793103 | 562.95\n2020-01-01 00:00:00+00  | 10     | 731.558894646 |  547.5\n2020-01-01 18:00:00+00  |  9     | 724.052854758 | 536.22\n2020-01-01 09:00:00+00  |  9     | 719.944816054 | 529.74\n2020-01-01 10:00:00+00  | 10     | 694.303472454 |  507.5\n2020-01-01 20:00:00+00  | 10     | 696.328870432 |  500.8\n2020-01-01 00:00:00+00  |  8     | 622.262145329 | 466.56\n2020-01-01 08:00:00+00  |  8     | 597.849434276 | 437.12\n2020-01-01 16:00:00+00  |  8     | 597.591488294 | 433.92\n2020-01-01 01:00:00+00  | 12     | 511.567512521 | 390.24\n ```\n I can see the pattern in my data again! The median was much better at dealing with outliers than `avg` was, and percentiles in general are much less noisy. This becomes even more obvious where we might want to measure the worst case scenario for users. So we might want to use the `max`, but often the 99th percentile value gives a better representation of the *likely* worst outcome for users than the max response time, which might be due to unrealistic parameters, an error, or some other non-representative condition. The maximum response time becomes something useful for engineers to investigate, ie to find errors or other weird outlier use cases, but less useful for, say, measuring overall user experience and how it changes over time. Both are useful for different circumstances, but often the 95th or 99th or other percentile outcome becomes the design parameter and what we measure success against.\n\n---\n### Why use *approximate* percentiles? <a id=\"why-approximate\"></a>\n\nOne reason that percentiles are less frequently used than, say, average, min, max or other measures of a distribution is that they are significantly more expensive to perform (in terms of cpu and memory) than traditional aggregates. This is because an exact computation of the percentile (using say, Postgres' [`percentile_cont`]() or [`percentile_disc`]() ) requires the full data set as an ordered list. This is unlike, say, the maximum where I can scan my data set and just keep the largest value I see, for percentiles I need to order the entire data set in order to find the 99th percentile or the 50th percentile etc. This also means that the aggregates are not partializable or parallelizable; there isn't a great form that will allow me to compute the exact percentile on part of my data and combine that with information from another part and give me an exact percentile back. I need all the data, ordered appropriately in order to calculate the exact result.\n\nThis is where approximation algorithms come into play: they allow for the calculation of a \"good enough\" percentile without using all of the data and ordering it before returning a result. There are multiple types of approximation algorithms, we've implemented two of them to start ([uddsketch]() and [tdigest]()), but if you're just getting started, we recommend trying out our [default implementation](), which uses the `uddsketch` implementation, but doesn't require twiddling of various knobs by the user. We believe this will be good enough for most cases, but if you run into an edge case or want different tradeoffs in terms of accuracy etc. we recommend reading [about the algorithms and tradeoffs below]() .\n\nLet's look back at our example from above and use our approximation algorithm alongside:\n\n```SQL\nSELECT\n    time_bucket('1 h'::interval, ts) as bucket,\n    api_id,\n    avg(response_time_ms),\n    percentile_disc(0.5) WITHIN GROUP (ORDER BY response_time_ms) as true_median,\n    approx_percentile(0.5, percentile_agg(response_time_ms))  as approx_median\nFROM response_times\nGROUP BY 1, 2\nORDER BY 5 DESC LIMIT 15;\n```\n```output, precision(2: 7)\n         bucket         | api_id |      avg      |  true_median  | approx_median\n------------------------+--------+---------------+---------------+---------------\n2020-01-01 00:00:00+00  | 12     | 993.878689655 | 751.68        | 764.998764437\n2020-01-01 12:00:00+00  | 12     |      948.4199 |  714.6        | 717.572650369\n2020-01-01 00:00:00+00  | 11     | 848.218549223 |    638        | 631.358694271\n2020-01-01 22:00:00+00  | 11     | 824.517045075 | 606.32        | 611.475044532\n2020-01-01 11:00:00+00  | 11     | 824.277392027 | 603.79        | 611.475044532\n2020-01-01 00:00:00+00  |  9     | 739.073793103 | 562.95        | 573.566636623\n2020-01-01 00:00:00+00  | 10     | 731.558894646 |  547.5        | 555.503056905\n2020-01-01 18:00:00+00  |  9     | 724.052854758 | 536.22        | 538.008361239\n2020-01-01 09:00:00+00  |  9     | 719.944816054 | 529.74        | 538.008361239\n2020-01-01 20:00:00+00  | 10     | 696.328870432 |  500.8        | 504.654521865\n2020-01-01 10:00:00+00  | 10     | 694.303472454 |  507.5        | 504.654521865\n2020-01-01 00:00:00+00  |  8     | 622.262145329 | 466.56        | 473.368454447\n2020-01-01 08:00:00+00  |  8     | 597.849434276 | 437.12        | 444.021967419\n2020-01-01 16:00:00+00  |  8     | 597.591488294 | 433.92        | 444.021967419\n2020-01-01 01:00:00+00  | 12     | 511.567512521 | 390.24        | 390.674211779\n```\nPretty darn close! We can definitely still see the patterns in the data. Note that the calling conventions are a bit different for ours, partially because it's no longer an [ordered set aggregate](), and partially because we use [two-step aggregation](), see the [API documentation]() below for exactly how to use.\n\nThe approximation algorithms can provide better performance than algorithms that need the whole sorted data set, especially on very large data sets that can't be easily sorted in memory. Not only that, but they are able to be incorporated into [continuous aggregates](), because they have partializable forms, can be used in [parallel]() and [partitionwise]() aggregation. They are used very frequently in continuous aggregates as that's where they give the largest benefit over the usual Postgres percentile algorithms, which can't be used at all because they require the entire ordered data set to function.\n\nLet's do this with our example, we can't use `percentile_disc` anymore as ordered set aggregates are not supported.\n\n```SQL , non-transactional, ignore-output\nCREATE MATERIALIZED VIEW response_times_hourly\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT\n    time_bucket('1 h'::interval, ts) as bucket,\n    api_id,\n    avg(response_time_ms),\n    percentile_agg(response_time_ms)\nFROM response_times\nGROUP BY 1, 2;\n```\nNote that we only do the aggregation step of our [two-step aggregation](), we'll save the accessor step for our selects from the view, and we'll start by just getting the same data as our previous example like so:\n\n```SQL\nSELECT\n    bucket,\n    api_id,\n    avg,\n    approx_percentile(0.5, percentile_agg) as approx_median\nFROM response_times_hourly\nORDER BY 4 DESC, 2, 1 LIMIT 15;\n```\n```output, precision(2: 7)\n         bucket         | api_id |      avg      | approx_median\n------------------------+--------+---------------+---------------\n2020-01-01 00:00:00+00  | 12     | 993.878689655 | 764.998764437\n2020-01-01 12:00:00+00  | 12     |      948.4199 | 717.572650369\n2020-01-01 00:00:00+00  | 11     | 848.218549223 | 631.358694271\n2020-01-01 11:00:00+00  | 11     | 824.277392027 | 611.475044532\n2020-01-01 22:00:00+00  | 11     | 824.517045075 | 611.475044532\n2020-01-01 00:00:00+00  |  9     | 739.073793103 | 573.566636623\n2020-01-01 00:00:00+00  | 10     | 731.558894646 | 555.503056905\n2020-01-01 09:00:00+00  |  9     | 719.944816054 | 538.008361239\n2020-01-01 18:00:00+00  |  9     | 724.052854758 | 538.008361239\n2020-01-01 10:00:00+00  | 10     | 694.303472454 | 504.654521865\n2020-01-01 20:00:00+00  | 10     | 696.328870432 | 504.654521865\n2020-01-01 00:00:00+00  |  8     | 622.262145329 | 473.368454447\n2020-01-01 08:00:00+00  |  8     | 597.849434276 | 444.021967419\n2020-01-01 16:00:00+00  |  8     | 597.591488294 | 444.021967419\n2020-01-01 00:00:00+00  |  6     | 500.610103448 | 390.674211779\n```\n\nSo, that's nifty, and much faster, especially for large data sets. But what's even cooler is I can do aggregates over the aggregates and speed those up, let's look at the median by `api_id`:\n```SQL\nSELECT\n    api_id,\n    approx_percentile(0.5, rollup(percentile_agg)) as approx_median\nFROM response_times_hourly\nGROUP BY api_id\nORDER BY api_id;\n```\n```output\n api_id | approx_median\n--------+---------------\n 1      | 54.5702804443\n 2      | 80.1171187405\n 3      | 103.491515519\n 4      | 91.0573557571\n 5      | 110.331520385\n 6      | 117.623597735\n 7      | 110.331520385\n 8      | 117.623597735\n 9      | 133.685458898\n10      | 117.623597735\n11      | 125.397626136\n12      | 133.685458898\n```\n\nYou'll notice that I didn't include the average response time here, that's because `avg` is not a [two-step aggregate](), and doesn't actually give you the average if you stack calls using it. But it turns out, we can derive the true average from the sketch we use to calculate the approximate percentiles! (We call that accessor function `mean` because there would otherwise be odd conflicts with `avg` in terms of how they're called).\n```SQL\nSELECT\n    api_id,\n    mean(rollup(percentile_agg)) as avg,\n    approx_percentile(0.5, rollup(percentile_agg)) as approx_median\nFROM response_times_hourly\nGROUP BY api_id\nORDER BY api_id;\n```\n```output, precision(1: 7)\n api_id |      avg      | approx_median\n--------+---------------+---------------\n 1      | 358.974815406 | 54.5702804443\n 2      | 116.208743234 | 80.1171187405\n 3      | 151.194417418 | 103.491515519\n 4      | 150.963527481 | 91.0573557571\n 5      | 180.906869604 | 110.331520385\n 6      | 202.234328036 | 117.623597735\n 7      | 203.056659681 | 110.331520385\n 8      | 210.823512283 | 117.623597735\n 9      | 250.775971756 | 133.685458898\n10      | 239.834855656 | 117.623597735\n11      | 267.750932477 | 125.397626136\n12      | 256.252763567 | 133.685458898\n```\n\nWe have several other accessor functions, including `error` which returns the maximum relative error for the percentile estimate, `num_vals` which returns the number of elements in the estimator, and perhaps the most interesting one, `approx_percentile_rank`, which gives the hypothetical percentile for a given value. Let's say we really don't want our apis to go over 1s in response time (1000 ms), we can use that to figure out what fraction of users waited over a second for each api:\n\n```SQL\nSELECT\n    api_id,\n    ((1 - approx_percentile_rank(1000, rollup(percentile_agg))) * 100)::numeric(6,2) as percent_over_1s\nFROM response_times_hourly\nGROUP BY api_id\nORDER BY api_id;\n```\n```output\n api_id | percent_over_1s\n--------+-----------------\n 1      | 0.07\n 2      | 0.00\n 3      | 0.00\n 4      | 0.40\n 5      | 1.56\n 6      | 2.54\n 7      | 2.87\n 8      | 3.30\n 9      | 4.56\n10      | 4.54\n11      | 5.90\n12      | 4.97\n```\n\n\n## API <a id=\"percentile-approx-api\"></a>\nAggregate Functions <a id=\"aggregate-functions\">\n> - [percentile_agg (point form)](#point-form)\n> - [rollup (summary form)](#summary-form)\n\nAccessor Functions <a id=\"accesor-functions\">\n\n> - [error](#error)\n> - [mean](#mean)\n> - [num_vals](#num-vals)\n> - [approx_percentile](#approx_percentile)\n> - [approx_percentile_rank](#approx_percentile-at-value)\n\n\n---\n## **percentile_agg (point form)** <a id=\"point-form\"></a>\n```SQL ,ignore\npercentile_agg(\n    value DOUBLE PRECISION\n) RETURNS UddSketch\n```\n\nThis is the default percentile aggregation function. Under the hood, it uses the [UddSketch algorithm](/docs/uddsketch.md) with 200 buckets and an initial max error of 0.001. This should be good for most common use cases of percentile approximation. For more advanced usage of the uddsketch algorithm or use cases for other percentile approximation algorithms see [advanced usage](#advanced-usage). This is the aggregation step of the [two-step aggregate](/docs/two-step_aggregation.md), it is usually used with the [approx_percentile()](#approx_percentile) accessor function in order to extract an approximate percentile, however it is in a form that can be re-aggregated using the [summary form](#summary-form) of the function and any of the other [accessor functions](#accessor-functions).\n\n\n### Required Arguments <a id=\"point-form-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `value` | `DOUBLE PRECISION` |  Column to aggregate.\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `percentile_agg` | `UddSketch` | A UddSketch object which may be passed to other percentile approximation APIs|\n\nBecause the `percentile_agg` function uses the [UddSketch algorithm](/docs/uddsketch.md), it returns the UddSketch data structure for use in further calls.\n<br>\n\n### Sample Usages <a id=\"point-form-examples\"></a>\n\nGet the approximate first percentile using the `percentile_agg()` point form plus the [`approx_percentile`](#approx_percentile) accessor function.\n```SQL\nSELECT\n    approx_percentile(0.01, percentile_agg(data))\nFROM generate_series(0, 100) data;\n```\n```output\napprox_percentile\n-------------------\n             0.999\n```\n\nThey are often used to create [continuous aggregates]() after which we can use multiple [accessors](#accessor-functions) for [retrospective analysis](/docs/two-step_aggregation.md#retrospective-analysis).\n\n```SQL ,ignore\nCREATE MATERIALIZED VIEW foo_hourly\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT\n    time_bucket('1 h'::interval, ts) as bucket,\n    percentile_agg(value) as pct_agg\nFROM foo\nGROUP BY 1;\n```\n---\n\n## **rollup (summary form)** <a id=\"summary-form\"></a>\n```SQL ,ignore\nrollup(\n    sketch uddsketch\n) RETURNS UddSketch\n```\n\nThis will combine multiple outputs from the [point form](#point-form) of the `percentile_agg()` function, this is especially useful for re-aggregation in the [continuous aggregate]() context (ie bucketing by a larger [`time_bucket`](), or re-grouping on other dimensions included in an aggregation).\n\n### Required Arguments <a id=\"summary-form-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `sketch` | `UddSketch` | The already constructed uddsketch from a previous [percentile_agg()](#point-form) call. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `uddsketch` | `UddSketch` | A UddSketch object which may be passed to other UddSketch APIs. |\n\nBecause the `percentile_agg` function uses the [UddSketch algorithm](/docs/uddsketch.md), `rollup` returns the UddSketch data structure for use in further calls.\n<br>\n\n### Sample Usages <a id=\"summary-form-examples\"></a>\nLet's presume we created the [continuous aggregate]() in the [point form example](#point-form-examples):\n\nWe can then rollup function to re-aggregate the results from the `foo_hourly` view and the [`approx_percentile`](#approx_percentile) accessor function to get the 95th and 99th percentiles over each day:\n\n```SQL , ignore\nSELECT\n    time_bucket('1 day'::interval, bucket) as bucket,\n    approx_percentile(0.95, rollup(pct_agg)) as p95,\n    approx_percentile(0.99, rollup(pct_agg)) as p99\nFROM foo_hourly\nGROUP BY 1;\n```\n\n---\n\n\n## **error** <a id=\"error\"></a>\n\n```SQL ,ignore\nerror(sketch UddSketch) RETURNS DOUBLE PRECISION\n```\n\nThis returns the maximum relative error that a percentile estimate will have (relative to the correct value). This means the actual value will fall in the range defined by `approx_percentile(sketch) +/- approx_percentile(sketch)*error(sketch)`.\n\n### Required Arguments <a id=\"error-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `sketch` | `UddSketch` | The sketch to determine the error of, usually from a [`percentile_agg()`](#aggregate-functions) call. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `error` | `DOUBLE PRECISION` | The maximum relative error of any percentile estimate. |\n<br>\n\n### Sample Usages <a id=\"error-examples\"></a>\n\n```SQL\nSELECT error(percentile_agg(data))\nFROM generate_series(0, 100) data;\n```\n```output\n error\n-------\n 0.001\n```\n\n---\n## **mean** <a id=\"mean\"></a>\n\n```SQL ,ignore\nmean(sketch UddSketch) RETURNS DOUBLE PRECISION\n```\n\nGet the exact average of all the values in the percentile estimate. (Percentiles returned are estimates, the average is exact.\n\n### Required Arguments <a id=\"mean-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `sketch` | `UddSketch` |  The sketch to extract the mean value from, usually from a [`percentile_agg()`](#aggregate-functions) call. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `mean` | `DOUBLE PRECISION` | The average of the values in the percentile estimate. |\n<br>\n\n### Sample Usage <a id=\"mean-examples\"></a>\n\n```SQL\nSELECT mean(percentile_agg(data))\nFROM generate_series(0, 100) data;\n```\n```output\n mean\n------\n 50\n```\n## **num_vals** <a id=\"num-vals\"></a>\n\n```SQL ,ignore\nnum_vals(sketch UddSketch) RETURNS DOUBLE PRECISION\n```\n\nGet the number of values contained in a percentile estimate.\n\n### Required Arguments <a id=\"num-vals-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `sketch` | `UddSketch` | The sketch to extract the number of values from, usually from a [`percentile_agg()`](#aggregate-functions) call. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `uddsketch_count` | `DOUBLE PRECISION` | The number of values in the percentile estimate |\n<br>\n\n### Sample Usage <a id=\"num-vals-examples\"></a>\n\n```SQL\nSELECT num_vals(percentile_agg(data))\nFROM generate_series(0, 100) data;\n```\n```output\n num_vals\n-----------\n       101\n```\n\n---\n---\n## **approx_percentile** <a id=\"approx_percentile\"></a>\n\n```SQL ,ignore\napprox_percentile(\n    percentile DOUBLE PRECISION,\n    sketch  uddsketch\n) RETURNS DOUBLE PRECISION\n```\n\nGet the approximate value at a percentile from a percentile estimate.\n\n### Required Arguments <a id=\"approx_percentile-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `approx_percentile` | `DOUBLE PRECISION` | The desired percentile (0.0-1.0) to approximate. |\n| `sketch` | `UddSketch` | The sketch to compute the approx_percentile on, usually from a [`percentile_agg()`](#aggregate-functions) call. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `approx_percentile` | `DOUBLE PRECISION` | The estimated value at the requested percentile. |\n<br>\n\n### Sample Usage <a id=\"approx_percentile-examples\"></a>\n\n```SQL\nSELECT\n    approx_percentile(0.01, percentile_agg(data))\nFROM generate_series(0, 100) data;\n```\n```output\napprox_percentile\n-------------------\n             0.999\n```\n\n---\n## **approx_percentile_rank** <a id=\"approx_percentile_rank\"></a>\n\n```SQL ,ignore\napprox_percentile_rank(\n    value DOUBLE PRECISION,\n    sketch UddSketch\n) RETURNS UddSketch\n```\n\nEstimate what percentile a given value would be located at in a UddSketch.\n\n### Required Arguments <a id=\"approx_percentile_rank-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `value` | `DOUBLE PRECISION` |  The value to estimate the percentile of. |\n| `sketch` | `UddSketch` | The sketch to compute the percentile on. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `approx_percentile_rank` | `DOUBLE PRECISION` | The estimated percentile associated with the provided value. |\n<br>\n\n### Sample Usage <a id=\"approx_percentile_rank-examples\"></a>\n\n```SQL\nSELECT\n    approx_percentile_rank(99, percentile_agg(data))\nFROM generate_series(0, 100) data;\n```\n```output\n approx_percentile_rank\n------------------------\n         0.985148514851\n```\n\n\n\n## Advanced Usage: Percentile Approximation Algorithms and How to Choose <a id=\"advanced-usage\"></a>\nWhile the simple `percentile_agg` interface will be sufficient for many users, we do provide more specific APIs for advanced users who want more control of how their percentile approximation is computed and how much space the intermediate representation uses.  We currently provide implementations of the following percentile approximation algorithms:\n\n- [T-Digest](/docs/tdigest.md) – This algorithm buckets data more aggressively toward the center of the quantile range, giving it greater accuracy near the tails (i.e. 0.001 or 0.995).\n- [UddSketch](/docs/uddsketch.md) – This algorithm uses exponentially sized buckets to guarantee the approximation falls within a known error range, relative to the true discrete percentile.\n\nThere are different tradeoffs that each algorithm makes, and different use cases where each will shine.  The doc pages above each link to the research papers fully detailing the algorithms if you want all the details.  However, at a higher level, here are some of the differences to consider when choosing an algorithm:\n1) First off, it's interesting to note that the formal definition for a percentile is actually impercise, and there are different methods for determining what the true percentile actually is.  In Postgres, given a target percentile 'p', `percentile_disc` will return the smallest element of a set such that 'p' percent of the set is less than that element, while `percentile_cont` will return an interpolated value between the two nearest matches for 'p'.  The difference here isn't usually that interesting in practice, but if it matters to your use case, then keep in mind that TDigest will approximate the continuous percentile while UddSketch provides an estimate of the discrete value.\n2) It's also important to consider the types of percentiles you're most interested in.  In particular, TDigest is optimized to trade off more accurate estimates at the extremes with weaker estimates near the median.  If your work flow involves estimating 99th percentiles, this is probably a good trade off.  However if you're more concerned about getting highly accurate median estimates, UddSketch is probably a better fit.\n3) UddSketch has a stable bucketing function, so it will always return the same quantile estimate for the same underlying data, regardless of how it is ordered or reaggregated.  TDigest, on the other hand, builds up incremental buckets based on the average of nearby points, which will result in (usually subtle) differences in estimates based on the same data, unless the order and batching of the aggregation is strictly controlled (which can be difficult to do in Postgres).  Therefore, if having stable estimates is important to you, UddSketch will likely be required.\n4) Trying to calculate precise error bars for TDigest can be difficult, especially when merging multiple subdigests into a larger one (this can come about either through summary aggregation or just parallelization of the normal point aggregate).  If being able to tightly characterize your error is important, UddSketch will likely be the desired algorithm.\n5) That being said, the fact that UddSketch uses exponential bucketing to provide a guaranteed relative error can cause some wildly varying absolute errors if the data set covers a large range.  For instance if the data is evenly distributed over the range [1,100], estimates at the high end of the percentile range would have about 100 times the absolute error of those at the low end of the range.  This gets much more extreme if the data range is [0,100].  If having a stable absolute error is important to your use case, consider TDigest.\n6) While both implementation will likely get smaller and/or faster with future optimizations, in general UddSketch will end up with a smaller memory footprint than TDigest, and a correspondingly smaller disk footprint for any continuous aggregates.  This is one of the main reasons that the default `percentile_agg` uses UddSketch, and is a pretty good reason to prefer that algorithm if your use case doesn't clearly benefit from TDigest.  Regardless of the algorithm, the best way to improve the accuracy of your percentile estimates is to increase the number of buckets, and UddSketch gives you more leeway to do so.\n"
  },
  {
    "path": "docs/release.md",
    "content": "# Release and build procedures\n\nWe build the timescaledb_toolkit extension using Cargo, but we have many\nhigher-level tasks in need of automation:\n\n- Build, lint, and test with particular flags in multiple environments\n- Extract SQL examples from documentation and test\n- Test upgrades\n- Installation\n- Publish a release\n- Make a container image to run all the above on\n\nThe rest of this document elaborates on each of those.  But first..\n\n## Dependency management\n\nIdeally, all dependencies would be specified in just one place.  But that's\nnot quite feasible.  Cargo.toml files capture the crate dependencies.\n\nThe rest are needed by the six shell scripts used to solve the above list\nof problems.  We configure those in `tools/dependencies.sh`.\n\n## Build, lint, and test\n\n`tools/build` is the relatively simple shell script that owns the cargo flags\nfor running clippy, running tests, installing, and testing upgrades.\nThe latter two are arguably out of place here.\n\nTesting upgrades is now handled by `testbin` (below), but the version here was\nuseful for Mac.  That has now degraded as it would need to support a third\npgrx...\n\nInstalling is only relevant for local development.\n\n## Extract SQL examples from documentation and test\n\n`tools/sql-doctester` is a Rust program which extracts example SQL programs\nand expected output from documentation, runs the programs, and asserts their\noutput matches what was expected.\n\nThe intent here is merely to prevent sample code from bitrotting, but some\nfunctionality is currently only tested here.\n\n## Test upgrades\n\nWe include in each release a set of scripts to upgrade an installation from a\nset of previous versions.  We test these upgrades by installing a supported\nold version, materializing some data, running the upgrade script, and\nasserting the extension can still load the old data.\n\n`tools/update-tester` is a Rust program which loads tests from `tests/update`\nto implement the materialize and verify steps.  It needs to know which version\neach function was stabilized in, and we store that information in\n`extension/src/stabilization_info.rs` (also used by post-install, see below).\n\n`tools/testbin` is a shell script that uses `update-tester` to test upgrades\nbetween released binaries (deb and rpm).\n\n## Installation\n\nInstallation is a two-step process currently duplicated in three places.\nThe two steps are:\n\n1. `cargo pgrx install --release` OR `cargo pgrx package`\n2. `tools/post-install`\n\nThese steps are repeated in:\n\n1. `Readme.md`\n2. `tools/build`\n3. `toolkit/package-deb.sh` and `toolkit/package-rpm.sh`\n\n`Readme.md` could simply recommend running `tools/build install`.\n\n`package-deb.sh` and `package-rpm.sh` could run `tools/build package` (which\ndoesn't yet exist).\n\n`cargo pgrx install` installs the extension into the directory specified by\n`pg_config`.  `cargo pgrx package` installs into a directory under\n`$CARGO_TARGET_DIR` where we pick it up and pack it into deb and rpm packages.\n\n`tools/post-install` performs miscellaneous install-time procedures:\n- finalize control file\n- rename `timescaledb_toolkit.so` to include the version number\n- generate update scripts\n\n`tools/post-install` needs to know which version each function was stabilized\nin, and we store that information in `extension/src/stabilization_info.rs`.\n\n## Publish a release\n\n`tools/release` automates all the steps of our release process.  We run it via\ngithub action (`.github/workflows/release.yml`).\n\n`tools/release` creates and pushes a release branch and tag, runs tests,\nstarts a package build, prepares the `main` branch for the next release, and\ncreates an issue so we don't forget some tasks not yet automated.\n\nThe package build happens in a different repository for reasons described in\ncomments at the top of `tools/release`.\n\nOver in that repository, we have `.github/workflows/toolkit-package.yml` which\nruns `toolkit/docker-run-package.sh` to build packages and\n`toolkit/upload-packages.sh` to upload them to PackageCloud.\n\n`toolkit/docker-run-package.sh` runs `toolkit/package-deb.sh` and\n`toolkit/package-rpm.sh` in various container images to build packags for\nthose platforms.  Which platforms we build for is controlled in the `yml`\naction file.\n\n### Usage:\n\n1. https://github.com/timescale/timescaledb-toolkit/actions/workflows/release.yml\n2. Click \"Run workflow\"\n3. Fill out the form\n4. Be sure to replace `-n` with `-push`!\n\nWe can replace this last one with a checkbox:\n- unchecked: run with neither `-n` nor `-push`\n- checked: run with `-push`\n\nThe script has three modes:\n\n- `-n`:  print what would be done without doing anything\n- `-push`:  do everything including pushing to Github and PackageCloud\n- neither:  do all the work (branch, edit, test, package, upgrade-test), but don't push anywhere\n\nThe third mode is the most useful but it is not available from the Github\naction.  Very sad.  We need to fix that.\n\n### Resume after failure\n\nUp until the packaging step, just rerun the release action after the problem\nis resolved.\n\nIf packages have been published, the choices are:\n- do the rest of what the script does manually\n- increment the patch revision (1.3.X) and start another release\n\nAn obvious improvement would be to teach `tools/release` to resume at a\nspecific step, something like `tools/release --start-at-step 7`.  It would\nneed to verify that the previous steps were actually done and bail out if not.\n\nOnce packaging is no longer asynchronous in the other repository,\n`tools/release` can simply be taught to figure out which steps are done all on\nits own, without an operator having to tell it where to resume.\n\n### Debugging\n\nWe run `tools/release` with the shell's `-x` option so it prints each command\nit runs.  We redirect the standard error stream to the standard output because\nDocker will otherwise separate them such that error messages may appear far\nfrom related output.\n\nSo, when something goes wrong, it is easy to pinpoint exactly which part of\nthe script failed and how.\n\nThings that can go wrong:\n\n#### Transient network hiccough\n\nThis can happen at almost any stage.  A simple retry might be the easiest way\nto see if the issue is transient.  If it's not, options are limited:\n- wait\n- complain, then wait\n\n#### cargo install cargo-edit\n\n- Is crates.io down?\n- Has cargo-edit vanished from crates.io?\n\n#### Install gh\n\n- The version we use is gone.  Find the latest and figure out whether all our\n  usage has been invalidated by incompatible changes.  Be careful!\n- Or, just squirrel away a copy of the old binary and keep rolling, until the\n  underlying APIs it uses break.\n- The checksum doesn't match.  Did they break it?  Why would they do such a\n  thing?  Were they hacked?  Probably should go ahead and update at this point.\n\n#### `extension/timescaledb_toolkit.control` problems\n\n`tools/release` edits this file, so it is very careful that the file looks the\nway it expects.  It is and should remain very picky.\n\nIf we've made some unexpected edits, it will complain.  If the edits were\nerroneous, fix them; else, you have to teach `tools/release` what you've done.\n\nOne of the things it checks is the `upgradeable_from` line.  Most importantly,\nit expects that patch releases are upgradeable from the previous version in\nthe same minor version (e.g. 1.3.1 is upgradeable from 1.3.0).\n\n#### `Changelog.md` problems\n\n`tools/release` ensures the version being released has Changelog.md entries.\n\nIt also requires some particular boiler-plate text at the top to know where to\nmake its edits.  The boiler-plate is arbitrary text for intended for\nconsumption by the development team.  If we change that text, `tools/release`\nneeds to know about it.\n\n#### Tests fail\n\nOh boy!\n\nTest output is logged.\n\n`tools/build test-extension` shouldn't fail since it already passed when the\nrelease commit was merged to master.\n\nYou're not trying to release a commit that didn't pass CI, are you?\n\nBut, the upgrade tests are being run for the first time!  So those might\nbreak.  We should run `tools/release --no-push` nightly.  In the mean time...\nto the debugger!\n\n#### git push fails\n\nWe've had branch permission problems before...\n\nIs the authentication token working?\n\n#### `gh` fails\n\nIs GitHub API struggling?\n\nIs the authentication token working?\n\nHas the packaging action in the `release-build-scripts` repository\ngone missing?\n\n## Make a container image to run all the above on\n\n`.github/workflows/toolkit-image.yml` configures the GitHub action which\nbuilds all our supported container images.\n\nOne image is special:  debian-11-amd64.  This is the one we run all our GitHub\nactions on.\n\n`docker/ci/Dockerfile` is the entry-point and it runs `docker/ci/setup.sh` to\ndo the work:\n\n- Create the build user\n- Install necessary build tools and libraries\n- Install postgresql and timescaledb\n- Install `gh` github command-line tool used by `tools/release`\n- Install Rust and PGRX\n- Pre-fetch toolkit's crate dependencies to minimize work done at CI time\n\n## Maintenance tasks\n\nSo, we've automated build and release!  ONCE AND FOR ALL.  Right?\n\nAs the great Balki Bartokomous often said:\nof course not; don't be ridiculous.\n\nThese are the sorts of things we have to do from time to time:\n\n- Update Rust.  It moves pretty fast.\n- Update PGRX.  It moves even faster.\n- Update other crates.  `cargo audit` and `cargo update` are our friends.\n- Update OS versions.  Labels such as `rockylinux:9` eventually point to\n  something different or disappear entirely.  The former actually surprised us\n  once already.\n\n### Things we update blindly\n\nWe install the latest version of these every time, so they may change in\nsurprising ways at inopportune times.\n\n- fpm:  It's a Ruby script with lots of dependencies and we install the latest\n  version and it bit us on the ass once already.  We use it because someone\n  set it up for us a long time ago and no one has had the chance to sit down\n  and figure out how to write an RPM spec file.  Shouldn't take more than a\n  few hours, just haven't done it...\n\n- postgresql:  We install the latest version of a fixed set of major versions,\n  so this should be very unlikely to break on us.  Listed for completeness.\n\n- timescaledb:  We test with their master branch nightly, so we should be\n  ahead of this one.\n\n### Unknown Unknowns\n\nlol\n\nThey're inevitable.  You just need a good nose for debugging.\n"
  },
  {
    "path": "docs/rolling_average_api_working.md",
    "content": "\n# Info dump on rolling average APIs #\n\nRolling averages are currently nasty to do with with timescaledb (user complaint https://news.ycombinator.com/item?id=27051005).  In our timevector API we will eventually provide a function like\n```SQL , ignore\nmoving_average(window => '30 minutes', slide => '5 minutes', data)\n```\nHowever, because set-returning aggregates cannot not exist in Postgres, this will not work outside of the timevector API. Currently, doing rolling average properly requires windowed aggregates. In base SQL it is a real PITA because you have to do sum and count separately and then divide them yourself.\n\n```SQL , ignore\nSELECT\n    time_bucket('5 minutes', time) as bucket,\n    sum(sum(value)) OVER thirty_minutes / sum(count(value)) OVER thirty_minutes as rolling_average\nFROM data\nGROUP BY 1\nWINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes', time) RANGE '30 minutes' PRECEDING);\n```\nIdeally, to do a thirty-minute rolling average every 5 minutes we would provide an API like:\n\n```SQL , ignore\nSELECT\n    time_bucket('5 minutes', time) as bucket,\n    rolling_average('5 minutes', value) OVER thirty_minutes\nFROM data\nGROUP BY bucket\nWINDOW thirty_minutes as (ORDER BY ts RANGE '30 minutes' PRECEDING);\n```\nHowever, this once again runs into postgres limitations: we need to aggregate over the `value` column in order for this to query to be correctly executed; the `rolling_average()` executes strictly after the `GROUP BY`, and will only see things within its 5-minute group. To fix this issue we need a separate aggregation step. First we'll aggregate the data into 5-minute summaries, then we'll re-aggregate over 30-minute windows of summaries\n```SQL , ignore\nSELECT\n    time_bucket('5 minutes'::interval, time) as bucket,\n    average(\n        rolling(stats_agg(value)) OVER thirty_minutes\n    )\nFROM foo\nGROUP BY bucket\nWINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING);\n```\nWhile we could create a dedicated `rolling_average()` function used like\n```SQL , ignore\nSELECT\n    time_bucket('5 minutes'::interval, time) as bucket,\n    rolling_average(stats_agg(value)) OVER thirty_minutes\nFROM foo\nGROUP BY bucket\nWINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING);\n```\nfor non-trivial cases, where you want to gather multiple statistics over the same data, this ends up significantly less readable, compare\n```SQL , ignore\nSELECT\n    time_bucket('5 minutes'::interval, ts) as bucket,\n    rolling_average(stats_agg(value)) OVER thirty_minutes,\n    rolling_stddev(stats_agg(value)) OVER thirty_minutes,\n    rolling_approx_percentile(0.1, percentile_agg(val1)) OVER thirty_minutes,\n    rolling_approx_percentile(0.9, percentile_agg(val1)) OVER thirty_minutes\nFROM foo\nGROUP BY 1\nWINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING);\n```\nto\n```SQL , ignore\nSELECT\n    bucket,\n    average(rolling_stats),\n    stddev(rolling_stats),\n    approx_percentile(0.1, rolling_percentile),\n    approx_percentile(0.9, rolling_percentile)\nFROM (\n    SELECT\n        time_bucket('5 minutes'::interval, ts) as bucket,\n        rolling(stats_agg(value)) OVER thirty_minutes as rolling_stats,\n        rolling(percentile_agg(value)) OVER thirty_minutes as rolling_percentile\n    FROM foo\n    GROUP BY 1\n    WINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING)\n) aggs;\n```\nsince in real world, and all our documentation, we expect to see multi-statistic queries, we plan to optimize for readability in this case, and have separate rollup and query steps.\n\nSeparating out the re-aggregation step also allows for more powerful composition, for instance:\n```SQL , ignore\nSELECT\n    bucket,\n    average(rolling_stats) as rolling_average,\n    average(rolling(rolling_stats) OVER (ORDER BY bucket)) AS cumulative_average,\n    average(rolling(rolling_stats) OVER ()) as full_set_average,\n    average(rolling_stats) / average(rolling(rolling_stats) OVER ()) as normalized_average\nFROM (\n    SELECT\n        time_bucket('5 minutes'::interval, ts) as bucket,\n        rolling(stats_agg(value)) OVER thirty_minutes as rolling_stats\n    FROM foo\n    GROUP BY 1\n    WINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING)\n) aggs;\n```\n\n\n### A note on style and semantics\n\n```SQL , ignore\nSELECT\n    bucket,\n    average(rolling_stats),\n    stddev(rolling_stats),\n    approx_percentile(0.1, rolling_percentile),\n    approx_percentile(0.9, rolling_percentile)\nFROM (\n    SELECT\n        time_bucket('5 minutes'::interval, ts) as bucket,\n        rolling(stats_agg(value)) OVER thirty_minutes as rolling_stats,\n        rolling(percentile_agg(value)) OVER thirty_minutes as rolling_percentile\n    FROM foo\n    GROUP BY 1\n    WINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING)\n) aggs;\n```\nis equivalent to\n\n```SQL , ignore\nWITH aggs as (\n    SELECT\n        time_bucket('5 minutes'::interval, ts) as bucket,\n        rolling(stats_agg(value)) OVER thirty_minutes as rolling_stats,\n        rolling(percentile_agg(value)) OVER thirty_minutes as rolling_percentile\n    FROM foo\n    GROUP BY 1\n    WINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING)\n)\nSELECT\n    bucket,\n    average(rolling_stats),\n    stddev(rolling_stats),\n    approx_percentile(0.1, rolling_percentile),\n    approx_percentile(0.9, rolling_percentile)\nFROM aggs;\n```\n\nwhich is also equivalent to, for understanding the order of operations here\n\n```SQL , ignore\nWITH aggs as (\n    SELECT\n        time_bucket('5 minutes'::interval, ts) as bucket,\n        stats_agg(value),\n        percentile_agg(value)\n    FROM foo\n    GROUP BY 1\n),\nrolling_aggs as (\n    SELECT\n        bucket\n        rolling(stats_agg) OVER thirty_minutes as rolling_stats,\n        rolling(percentile_agg) OVER thirty_minutes as rolling_percentile\n    FROM aggs\n    WINDOW thirty_minutes as (ORDER BY bucket RANGE '30 minutes' PRECEDING)\n)\nSELECT\n    bucket,\n    average(rolling_stats),\n    stddev(rolling_stats),\n    approx_percentile(0.1, rolling_percentile),\n    approx_percentile(0.9, rolling_percentile)\nFROM rolling_aggs;\n```\n\nwhich is also equivalent to:\n\n```SQL , ignore\nSELECT\n    bucket,\n    average(rolling_stats),\n    stddev(rolling_stats),\n    approx_percentile(0.1, rolling_percentile),\n    approx_percentile(0.9, rolling_percentile)\nFROM (\n    SELECT\n        bucket,\n        rolling(stats_agg) OVER thirty_minutes as rolling_stats,\n        rolling(percentile_agg) OVER thirty_minutes as rolling_percentile\n    FROM (\n        SELECT\n            time_bucket('5 minutes'::interval, ts) as bucket,\n            stats_agg(value),\n            percentile_agg(value)\n        FROM foo\n        GROUP BY 1\n    ) aggs\n    WINDOW thirty_minutes as (ORDER BY time_bucket('5 minutes'::interval, ts) RANGE '30 minutes' PRECEDING)\n) rolling_aggs;\n```"
  },
  {
    "path": "docs/state_agg.md",
    "content": "# State Aggregation [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes)\n\n# Test table\n\nExamples below are tested against the following tables:\n\n```SQL ,non-transactional\nSET TIME ZONE 'UTC';\nCREATE TABLE states_test(ts TIMESTAMPTZ, state TEXT);\nINSERT INTO states_test VALUES\n    ('2020-01-01 00:00:00+00', 'START'),\n    ('2020-01-01 00:00:11+00', 'OK'),\n    ('2020-01-01 00:01:00+00', 'ERROR'),\n    ('2020-01-01 00:01:03+00', 'OK'),\n    ('2020-01-01 00:02:00+00', 'STOP');\nCREATE TABLE states_test_2(ts TIMESTAMPTZ, state TEXT);\nINSERT INTO states_test_2 VALUES\n    ('2019-12-31 00:00:00+00', 'START'),\n    ('2019-12-31 00:00:11+00', 'OK'),\n    ('2019-12-31 00:02:00+00', 'STOP'),\n    ('2019-12-31 00:01:03+00', 'OK');\nCREATE TABLE states_test_3(ts TIMESTAMPTZ, state TEXT);\nINSERT INTO states_test_3 VALUES\n    ('2019-12-31 00:00:11+00', 'UNUSED'),\n    ('2019-12-31 00:01:00+00', 'START');\nCREATE TABLE states_test_4(ts TIMESTAMPTZ, state BIGINT);\nINSERT INTO states_test_4 VALUES\n    ('2020-01-01 00:00:00+00', 4),\n    ('2020-01-01 00:00:11+00', 51351),\n    ('2020-01-01 00:01:00+00', 2),\n    ('2020-01-01 00:01:03+00', 51351),\n    ('2020-01-01 00:02:00+00', -9);\nCREATE TABLE states_test_5(ts TIMESTAMPTZ, state BIGINT); -- states_test with integer states\nINSERT INTO states_test_5 VALUES\n    ('2020-01-01 00:00:00+00', 4),\n    ('2020-01-01 00:00:11+00', 51351),\n    ('2020-01-01 00:01:00+00', 2),\n    ('2020-01-01 00:02:03+00', 51351),\n    ('2020-01-01 00:02:05+00', -9);\nCREATE TABLE states_test_6(ts TIMESTAMPTZ, state BIGINT); -- states_test_3 with integer states\nINSERT INTO states_test_6 VALUES\n    ('2019-12-31 00:00:11+00', 456789),\n    ('2019-12-31 00:01:00+00', 4);\n```\n\n## Functions\n\n### duration_in\n\nCompute the amount of time spent in a state as INTERVAL.\n\n```SQL\nSELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'ERROR') FROM states_test;\n```\n```output\n interval\n----------\n 00:00:03\n```\n```SQL\nSELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 2) FROM states_test_4;\n```\n```output\n interval\n----------\n 00:00:03\n```\n\nExtract as number of seconds:\n\n```SQL\nSELECT\n  EXTRACT(epoch FROM\n    toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'ERROR')\n  )::INTEGER\nFROM states_test;\n```\n```output\n seconds\n---------\n       3\n```\n\n#### duration_in for a range\n```SQL\nSELECT duration_in(state_agg(ts, state), 'OK', '2020-01-01 00:01:00+00', '2 days') FROM states_test;\n```\n```output\n duration_in\n-------------\n 00:00:57\n```\n```SQL\nSELECT duration_in(state_agg(ts, state), 'OK', '2020-01-01 00:01:00+00', NULL) FROM states_test;\n```\n```output\n duration_in\n-------------\n 00:00:57\n```\n```SQL\nSELECT duration_in(state_agg(ts, state), 'OK', '2020-01-01 00:01:00+00') FROM states_test;\n```\n```output\n duration_in\n-------------\n 00:00:57\n```\n```SQL\nSELECT duration_in(state_agg(ts, state), 51351, '2020-01-01 00:01:00+00', '2 days') FROM states_test_4;\n```\n```output\n duration_in\n-------------\n 00:00:57\n```\n```SQL\nSELECT duration_in(state_agg(ts, state), 51351, '2020-01-01 00:01:00+00', NULL) FROM states_test_4;\n```\n```output\n duration_in\n-------------\n 00:00:57\n```\n\n```SQL\nSELECT duration_in(state_agg(ts, state), 'OK', '2020-01-01 00:00:15+00', '30 seconds') FROM states_test;\n```\n```output\n duration_in\n-------------\n 00:00:30\n```\n\n```SQL\nSELECT duration_in(state_agg(ts, state), 51351, '2020-01-01 00:00:15+00', '1 minute 1 second') FROM states_test_4;\n```\n```output\n duration_in\n-------------\n 00:00:58\n```\n\n```SQL\nSELECT duration_in(state_agg(ts, state), 'OK', '2020-01-01 00:00:15+00', '1 minute 1 second') FROM states_test;\n```\n```output\n duration_in\n-------------\n 00:00:58\n```\n\n```SQL\nSELECT (SELECT state_agg(ts, state) FROM states_test) -> duration_in('OK'::text, '2020-01-01 00:00:15+00', '1 minute 1 second');\n```\n```output\n ?column?\n-------------\n 00:00:58\n```\n\n```SQL\nSELECT (SELECT state_agg(ts, state) FROM states_test) -> duration_in('OK');\n```\n```output\n ?column?\n-------------\n 00:01:46\n```\n\n### into_values\n\n```SQL\nSELECT state, duration FROM toolkit_experimental.into_values(\n    (SELECT toolkit_experimental.compact_state_agg(ts, state) FROM states_test))\n    ORDER BY state, duration;\n```\n```output\n state | duration\n-------+-----------\n ERROR |  00:00:03\n OK    |  00:01:46\n START |  00:00:11\n STOP  |  00:00:00\n```\n```SQL\nSELECT state, duration FROM into_int_values(\n    (SELECT state_agg(ts, state) FROM states_test_4))\n    ORDER BY state, duration;\n```\n```output\n state | duration\n-------+-----------\n   -9 |  00:00:00\n    2 |  00:00:03\n    4 |  00:00:11\n51351 |  00:01:46\n```\n```SQL\nSELECT (state_agg(ts, state) -> into_values()).* FROM states_test ORDER BY state;\n```\n```output\n state | duration\n-------+----------\n ERROR | 00:00:03\n OK    | 00:01:46\n START | 00:00:11\n STOP  | 00:00:00\n```\n\n### state_timeline\n\n```SQL\nSELECT (state_agg(ts, state) -> state_timeline()).* FROM states_test;\n```\n```output\n state |       start_time       |        end_time\n-------+------------------------+------------------------\n START | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:11+00\n OK    | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n ERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n OK    | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP  | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT state, start_time, end_time FROM state_timeline(\n    (SELECT state_agg(ts, state) FROM states_test))\n    ORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\nSTART | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:11+00\n   OK | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\nERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n   OK | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT state, start_time, end_time FROM state_int_timeline(\n    (SELECT state_agg(ts, state) FROM states_test_4))\n    ORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\n    4 | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:11+00\n51351 | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n    2 | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n51351 | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n   -9 | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n\n```SQL\nSELECT state, start_time, end_time FROM state_timeline(\n    (SELECT state_agg(ts, state) FROM states_test_2))\n    ORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\nSTART | 2019-12-31 00:00:00+00 | 2019-12-31 00:00:11+00\n   OK | 2019-12-31 00:00:11+00 | 2019-12-31 00:02:00+00\n STOP | 2019-12-31 00:02:00+00 | 2019-12-31 00:02:00+00\n```\n\n### state_in\n\n```SQL\nSELECT state_at(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2020-01-01 00:01:02+00'\n);\n```\n```output\n state_at\n----------\n ERROR\n```\n```SQL\nSELECT state_at_int(\n    (SELECT state_agg(ts, state) FROM states_test_5),\n    '2020-01-01 00:01:02+00'\n);\n```\n```output\n state_at\n----------\n 2\n```\n```SQL\nSELECT state_at(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2020-01-01 00:01:00+00'\n);\n```\n```output\n state_at\n----------\n ERROR\n```\n```SQL\nSELECT state_at(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2020-01-01 00:00:05+00'\n);\n```\n```output\n state_at\n----------\n START\n```\n```SQL\nSELECT state_at(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2020-01-01 00:00:00+00'\n);\n```\n```output\n state_at\n----------\n START\n```\n```SQL\nSELECT state_at(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2019-12-31 23:59:59.999999+00'\n);\n```\n```output\n state_at\n----------\n \n```\n```SQL\nSELECT state_at(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2025-01-01 00:00:00+00'\n);\n```\n```output\n state_at\n----------\n STOP\n```\n```SQL\nSELECT (SELECT state_agg(ts, state) FROM states_test) -> state_at('2025-01-01 00:00:00+00');\n```\n```output\n ?column?\n----------\n STOP\n```\n\n## state_periods\n\n```SQL\nSELECT start_time, end_time\nFROM state_periods(\n    (SELECT state_agg(ts, state) FROM states_test),\n    'OK'\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT ((SELECT state_agg(ts, state) FROM states_test) -> state_periods('OK')).*;\n```\n```output\n       start_time       |        end_time\n------------------------+------------------------\n 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT start_time, end_time\nFROM state_periods(\n    (SELECT state_agg(ts, state) FROM states_test_4),\n    51351\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT start_time, end_time\nFROM state_periods(\n    (SELECT state_agg(ts, state) FROM states_test),\n    'ANYTHING'\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n```\n\n## interpolated_state_timeline\n\n```SQL\nSELECT state, start_time, end_time FROM interpolated_state_timeline(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_3)\n)\nORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\nSTART | 2019-12-31 00:00:00+00 | 2020-01-01 00:00:11+00\n   OK | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\nERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n   OK | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT ((SELECT state_agg(ts, state) FROM states_test) -> interpolated_state_timeline(\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_3)\n)).*\nORDER BY start_time;\n```\n```output\n state |       start_time       |        end_time\n-------+------------------------+------------------------\n START | 2019-12-31 00:00:00+00 | 2020-01-01 00:00:11+00\n OK    | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n ERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n OK    | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP  | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT state, start_time, end_time FROM interpolated_state_int_timeline(\n    (SELECT state_agg(ts, state) FROM states_test_5),\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_6)\n)\nORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\n    4 | 2019-12-31 00:00:00+00 | 2020-01-01 00:00:11+00\n51351 | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n    2 | 2020-01-01 00:01:00+00 | 2020-01-01 00:02:03+00\n51351 | 2020-01-01 00:02:03+00 | 2020-01-01 00:02:05+00\n   -9 | 2020-01-01 00:02:05+00 | 2020-01-01 00:02:05+00\n```\n\n```SQL\nSELECT state, start_time, end_time FROM interpolated_state_timeline(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2019-12-31', '5 days',\n    (SELECT state_agg(ts, state) FROM states_test_3)\n)\nORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\nSTART | 2019-12-31 00:00:00+00 | 2020-01-01 00:00:11+00\n   OK | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\nERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n   OK | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP | 2020-01-01 00:02:00+00 | 2020-01-05 00:00:00+00\n```\n\n```SQL\nSELECT state, start_time, end_time FROM interpolated_state_timeline(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_2)\n)\nORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\n STOP | 2019-12-31 00:00:00+00 | 2020-01-01 00:00:00+00\nSTART | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:11+00\n   OK | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\nERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n   OK | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT state, start_time, end_time FROM interpolated_state_timeline(\n    (SELECT state_agg(ts, state) FROM states_test),\n    '2019-12-31', '5 days',\n    (SELECT state_agg(ts, state) FROM states_test_2)\n)\nORDER BY start_time;\n```\n```output\nstate | start_time             | end_time\n------+------------------------+-----------------------\n STOP | 2019-12-31 00:00:00+00 | 2020-01-01 00:00:00+00\nSTART | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:11+00\n   OK | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\nERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n   OK | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP | 2020-01-01 00:02:00+00 | 2020-01-05 00:00:00+00\n```\n\n```SQL\nSELECT (state_agg(ts, state) -> state_periods('OK')).* FROM states_test;\n```\n```output\n       start_time       |        end_time\n------------------------+------------------------\n 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n```\n\n## interpolated_state_periods\n\n```SQL\nSELECT start_time, end_time FROM interpolated_state_periods(\n    (SELECT state_agg(ts, state) FROM states_test),\n    'OK',\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_3)\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT ((SELECT state_agg(ts, state) FROM states_test) -> interpolated_state_periods(\n    'OK',\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_3)\n)).*\nORDER BY start_time;\n```\n```output\n       start_time       |        end_time\n------------------------+------------------------\n 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT start_time, end_time FROM interpolated_state_periods(\n    (SELECT state_agg(ts, state) FROM states_test),\n    'START',\n    '2019-12-31', '5 days',\n    (SELECT state_agg(ts, state) FROM states_test_3)\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2019-12-31 00:00:00+00 | 2020-01-01 00:00:11+00\n```\n\n```SQL\nSELECT start_time, end_time FROM interpolated_state_periods(\n    (SELECT state_agg(ts, state) FROM states_test_5),\n    4,\n    '2019-12-31', '5 days',\n    (SELECT state_agg(ts, state) FROM states_test_6)\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2019-12-31 00:00:00+00 | 2020-01-01 00:00:11+00\n```\n\n```SQL\nSELECT start_time, end_time FROM interpolated_state_periods(\n    (SELECT state_agg(ts, state) FROM states_test),\n    'STOP',\n    '2019-12-31', '1 days',\n    (SELECT state_agg(ts, state) FROM states_test_2)\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2019-12-31 00:00:00+00 | 2020-01-01 00:00:00+00\n2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n\n```SQL\nSELECT start_time, end_time FROM interpolated_state_periods(\n    (SELECT state_agg(ts, state) FROM states_test),\n    'STOP',\n    '2019-12-31', '5 days',\n    (SELECT state_agg(ts, state) FROM states_test_2)\n)\nORDER BY start_time;\n```\n```output\nstart_time             | end_time\n-----------------------+-----------------------\n2019-12-31 00:00:00+00 | 2020-01-01 00:00:00+00\n2020-01-01 00:02:00+00 | 2020-01-05 00:00:00+00\n```\n\n## rollup\n\n```SQL\nWITH buckets AS (SELECT\n    date_trunc('minute', ts) as dt,\n    toolkit_experimental.compact_state_agg(ts, state) AS sa\nFROM states_test\nGROUP BY date_trunc('minute', ts))\nSELECT toolkit_experimental.duration_in(\n    toolkit_experimental.rollup(buckets.sa),\n    'START'\n)\nFROM buckets;\n```\n```output\n interval\n----------\n 00:00:11\n```\n\n```SQL\nWITH buckets AS (SELECT\n    date_trunc('minute', ts) as dt,\n    toolkit_experimental.compact_state_agg(ts, state) AS sa\nFROM states_test\nGROUP BY date_trunc('minute', ts))\nSELECT toolkit_experimental.duration_in(\n    toolkit_experimental.rollup(buckets.sa),\n    'OK'\n)\nFROM buckets;\n```\n```output\n interval\n----------\n 00:01:46\n```\n\n```SQL\nWITH buckets AS (SELECT\n    date_trunc('minute', ts) as dt,\n    state_agg(ts, state) AS sa\nFROM states_test\nGROUP BY date_trunc('minute', ts))\nSELECT state_timeline(\n    rollup(buckets.sa)\n)\nFROM buckets;\n```\n```output\n                      state_timeline\n-----------------------------------------------------------\n(START,\"2020-01-01 00:00:00+00\",\"2020-01-01 00:00:11+00\")\n   (OK,\"2020-01-01 00:00:11+00\",\"2020-01-01 00:01:00+00\")\n(ERROR,\"2020-01-01 00:01:00+00\",\"2020-01-01 00:01:03+00\")\n   (OK,\"2020-01-01 00:01:03+00\",\"2020-01-01 00:02:00+00\")\n (STOP,\"2020-01-01 00:02:00+00\",\"2020-01-01 00:02:00+00\")\n```\n\n```SQL\nWITH buckets AS (SELECT\n    date_trunc('minute', ts) as dt,\n    state_agg(ts, state) AS sa\nFROM states_test\nGROUP BY date_trunc('minute', ts)\nHAVING date_trunc('minute', ts) != '2020-01-01 00:01:00+00'::timestamptz)\nSELECT state_timeline(\n    rollup(buckets.sa)\n)\nFROM buckets;\n```\n```output\n                      state_timeline\n-----------------------------------------------------------\n(START,\"2020-01-01 00:00:00+00\",\"2020-01-01 00:00:11+00\")\n   (OK,\"2020-01-01 00:00:11+00\",\"2020-01-01 00:02:00+00\")\n (STOP,\"2020-01-01 00:02:00+00\",\"2020-01-01 00:02:00+00\")\n```\n\n```SQL\nWITH buckets AS (SELECT\n    date_trunc('minute', ts) as dt,\n    state_agg(ts, state) AS sa\nFROM states_test_5\nGROUP BY date_trunc('minute', ts)\nHAVING date_trunc('minute', ts) != '2020-01-01 00:01:00+00'::timestamptz)\nSELECT state_int_timeline(\n    rollup(buckets.sa)\n)\nFROM buckets;\n```\n```output\n                      state_timeline\n-----------------------------------------------------------\n    (4,\"2020-01-01 00:00:00+00\",\"2020-01-01 00:00:11+00\")\n(51351,\"2020-01-01 00:00:11+00\",\"2020-01-01 00:02:05+00\")\n   (-9,\"2020-01-01 00:02:05+00\",\"2020-01-01 00:02:05+00\")\n```\n\n## With continuous aggregate\n\n```SQL ,non-transactional,ignore-output\nCREATE TABLE email_status (\n  id BIGINT,\n  ts TIMESTAMPTZ,\n  status TEXT\n);\nSELECT create_hypertable('email_status','ts');\n\nINSERT INTO email_status(\"ts\", \"id\", \"status\")\nVALUES\n('2022-01-11 11:51:12',1,'draft'),\n('2022-01-11 11:53:23',1,'queued'),\n('2022-01-11 11:57:46',1,'sending'),\n('2022-01-11 11:57:50',1,'sent'),\n('2022-01-11 11:52:12',2,'draft'),\n('2022-01-11 11:58:23',2,'queued'),\n('2022-01-11 12:00:46',2,'sending'),\n('2022-01-11 12:01:03',2,'bounced');\n```\n\n```SQL ,non-transactional,ignore-output\nCREATE MATERIALIZED VIEW sa\nWITH (timescaledb.continuous, timescaledb.materialized_only=false) AS\nSELECT time_bucket('1 minute'::interval, ts) AS bucket,\n  id,\n  state_agg(ts, status) AS agg\nFROM email_status\nGROUP BY bucket, id;\n```\n\n```SQL\nSELECT rollup(agg) -> duration_in('draft') FROM sa WHERE id = 1;\n```\n```output\n ?column?\n----------\n 00:02:11\n```\n\n```SQL\nSELECT (state_timeline(rollup(agg))).* FROM sa WHERE id = 2;\n```\n```output\n  state  |       start_time       |        end_time\n---------+------------------------+------------------------\n draft   | 2022-01-11 11:52:12+00 | 2022-01-11 11:58:23+00\n queued  | 2022-01-11 11:58:23+00 | 2022-01-11 12:00:46+00\n sending | 2022-01-11 12:00:46+00 | 2022-01-11 12:01:03+00\n bounced | 2022-01-11 12:01:03+00 | 2022-01-11 12:01:03+00\n```\n"
  },
  {
    "path": "docs/stats_agg.md",
    "content": "# Statistical Aggregates\n\n## Common 1-D Statistical Functions\n- `average`\n- `sum`\n- `num_vals`\n- `stddev`(population and sample)\n- `variance` (population and sample )\n- `skewness`\n- `kurtosis`\n\n## 2-D Statistical Regression Functions\n- `slope`\n- `intercept`\n- `x_intercept`\n- `corr` (correlation coefficient)\n- `covariance` (population  and sample)\n- `skewness`\n- `kurtosis`\n- `determination_coeff`\n\nIn order to make common statistical aggregates easier to work with in window functions and continuous aggregates, Toolkit provides common statistical aggregates in a slightly different form than  otherwise available in PostgreSQL/TimescaleDB. They are re-implemented within the [two-step aggregates framework](docs/two-step_aggregation.md)which exposes a summary form to the user which can then have multiple accessors. \n\n```SQL, non-transactional\nCREATE TABLE foo (\n    t timestamptz,\n    x DOUBLE PRECISION,\n    y DOUBLE PRECISION\n);\n```\n\nIn order to run any of these statistical functions you must first perform the `stats_agg` aggregate with either one or two variables, following the general SQL framework for these things, when being used for statistical regression with two dimensions, the dependent variable comes first and the independent variable second, ie:\n\n```SQL, ignore-output\nSELECT stats_agg(y, x) FROM foo;\n```\n\nAs with other aggregates in the Toolkit, you can use any of the accessors on the results of the aggregation, so: \n\n```SQL, ignore-output\nSELECT average(\n    stats_agg(x)\n) FROM foo;\n```\nwill give you the average of column `x`. While this is slightly more complex for the simple case, many of the results of these aggregates are not combinable in their final forms, the output of the `stats_agg` aggregate is combinable, which means we can do tumbling window aggregates with them and re-combine them when they are used in continuous aggregates. \n\nIn the 2-D case, you can access single variable statistics by calling the function with `_x` or `_y` like so:\n\n```SQL, ignore-output\nSELECT average_x(\n    stats_agg(y, x)\n) FROM foo;\n```\n\nStatistics involving both variables (the ones only available in the 2-D case) are called normally:\n```SQL, ignore-output\nSELECT slope(\n    stats_agg(y, x)\n) FROM foo;\n```\n\nFor those statistics which have variants for either the sample or population we have made these accessible via a separate variable ie:\n\n```SQL, ignore-output\nSELECT covariance(\n    stats_agg(y, x),\n    'population'\n) FROM foo;\n```\n\nThe default for all of these is 'population' (the abbreviations 'pop' and 'samp' are also acceptable). The default means the function may also be called without the second argument, like so:\n\n```SQL, ignore-output\nSELECT covariance(\n    stats_agg(y, x)\n) FROM foo;\n```\n\nWhich will still return the population covariance.\n\n\nThis is a minimum working version of the documentation for now, another working document can be found [here](docs/rolling_average_api_working.md), which goes into the window function usecase and some of the reasoning behind our naming decisions. Please feel free to open issues or discussions if you have questions or comments on the current API. We will further develop the documentation as we stabilize these functions over the coming releases. \n"
  },
  {
    "path": "docs/tdigest.md",
    "content": "# T-Digest\n\n> [Description](#tdigest-description)<br>\n> [Details](#tdigest-details)<br>\n> [Example](#tdigest-example)<br>\n> [Continuous Aggregate Example](#tdigest-cagg-example)<br>\n> [API](#tdigest-api)\n\n## Description <a id=\"tdigest-description\"></a>\n\nTimescaleDB Toolkit provides an implementation of the [t-digest data structure](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf) for quantile approximations.  A t-digest is a space efficient aggregation which provides increased resolution at the edges of the distribution.  This allows for more accurate estimates of extreme quantiles than traditional methods.\n\n## Details <a id=\"tdigest-details\"></a>\n\nTimescale's t-digest is implemented as an aggregate function in PostgreSQL.  They do not support moving-aggregate mode, and are not ordered-set aggregates.  Presently they are restricted to float values, but the goal is to make them polymorphic.  They are partializable and are good candidates for [continuous aggregation](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/).\n\nOne additional thing to note about TDigests is that they are somewhat dependent on the order of inputs.  The percentile approximations should be nearly equal for the same underlying data, especially at the extremes of the quantile range where the TDigest is inherently more accurate, they are unlikely to be identical if built in a different order.  While this should have little effect on the accuracy of the estimates, it is worth noting that repeating the creation of the TDigest might have subtle differences if the call is being parallelized by Postgres.  Similarly, building a TDigest by combining several subdigests using the [summary aggregate](#tdigest-summary) is likely to produce a subtley different result than combining all of the underlying data using a single [point aggregate](#tdigest).\n\n## Usage Example <a id=\"tdigest-example\"></a>\n\nFor this example we're going to start with a table containing some NOAA weather data for a few weather stations across the US over the past 20 years.\n\n```SQL ,ignore\n\\d weather;\n```\n```\n                         Table \"public.weather\"\n Column  |            Type             | Collation | Nullable | Default\n---------+-----------------------------+-----------+----------+---------\n station | text                        |           |          |\n name    | text                        |           |          |\n date    | timestamp without time zone |           |          |\n prcp    | double precision            |           |          |\n snow    | double precision            |           |          |\n tavg    | double precision            |           |          |\n tmax    | double precision            |           |          |\n tmin    | double precision            |           |          |\n```\n\nNow let's create some t-digests for our different stations and verify that they're receiving data.\n\n```SQL ,ignore\nCREATE VIEW high_temp AS\n    SELECT name, tdigest(100, tmax)\n    FROM weather\n    GROUP BY name;\n\nSELECT\n    name,\n    num_vals(tdigest)\nFROM high_temp;\n```\n```\n                 name                  | num_vals\n---------------------------------------+-----------\n PORTLAND INTERNATIONAL AIRPORT, OR US |      7671\n LITCHFIELD PARK, AZ US                |      5881\n NY CITY CENTRAL PARK, NY US           |      7671\n MIAMI INTERNATIONAL AIRPORT, FL US    |      7671\n(4 rows)\n```\n\nWe can then check to see the 99.5 percentile high temperature for each location.\n```SQL ,ignore\nSELECT\n    name,\n    approx_percentile(0.995, tdigest)\nFROM high_temp;\n```\n```\n                 name                  |           quantile\n---------------------------------------+--------------------\n PORTLAND INTERNATIONAL AIRPORT, OR US |   98.4390837104072\n LITCHFIELD PARK, AZ US                | 114.97809722222223\n NY CITY CENTRAL PARK, NY US           |  95.86391321044545\n MIAMI INTERNATIONAL AIRPORT, FL US    |  95.04283854166665\n(4 rows)\n```\nOr even check to see what quantile 90F would fall at in each city.\n```SQL ,ignore\nSELECT\n    name,\n    approx_percentile_rank(90.0, tdigest)\nFROM high_temp;\n```\n```\n                 name                  |  approx_percentile_rank\n---------------------------------------+--------------------\n PORTLAND INTERNATIONAL AIRPORT, OR US | 0.9609990016734108\n LITCHFIELD PARK, AZ US                | 0.5531621580122781\n NY CITY CENTRAL PARK, NY US           | 0.9657150306348585\n MIAMI INTERNATIONAL AIRPORT, FL US    | 0.8093468908877591\n(4 rows)\n```\n\n## Example Using TimeScale Continuous Aggregates (tdigest-cagg-example)\nTimescale [continuous aggregates](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/)\nprovide an easy way to keep a tdigest up to date as more data is added to a table.  The following example\nshows how this might look in practice.  The first step is to create a Timescale hypertable to store our data.\n\n```SQL ,non-transactional,ignore-output\nSET TIME ZONE 'UTC';\nCREATE TABLE test(time TIMESTAMPTZ, value DOUBLE PRECISION);\nSELECT create_hypertable('test', 'time');\n```\n\nNext a materialized view with the timescaledb.continuous property is added.  This will automatically keep itself,\nincluding the tdigest in this case, up to date as data is added to the table.\n```SQL ,non-transactional,ignore-output\nCREATE MATERIALIZED VIEW weekly_sketch\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT\n    time_bucket('7 day'::interval, time) as week,\n    tdigest(100, value) as digest\nFROM test\nGROUP BY time_bucket('7 day'::interval, time);\n```\n\nNext a utility function, `generate_periodic_normal_series`, is called to generate some data.  When called in\nthis manner the function will return 28 days worth of data points spaced 10 minutes apart.  These points are\ngenerate by adding a random point (with a normal distribution and standard deviation of 100) to a sine wave\nwhich oscilates between 900 and 1100 over the period of a day.\n```SQL ,non-transactional\nINSERT INTO test\n    SELECT time, value\n    FROM toolkit_experimental.generate_periodic_normal_series('2020-01-01 UTC'::timestamptz, rng_seed => 543643);\n```\n```\nINSERT 0 4032\n```\n\n<div hidden>\nRefresh to make sure we're exercising our serialization path.\n```SQL ,non-tranactional,ignore-output\nCALL refresh_continuous_aggregate('weekly_sketch', NULL, NULL);\n```\n</div>\n\nFinally, a query is run over the aggregate to see various approximate percentiles from different weeks.\n```SQL,ignore\nSELECT\n    week,\n    approx_percentile(0.01, digest) AS low,\n    approx_percentile(0.5, digest) AS mid,\n    approx_percentile(0.99, digest) AS high\nFROM weekly_sketch\nORDER BY week;\n```\n```ignore\n         week          |        low        |        mid         |        high\n-----------------------+-------------------+--------------------+--------------------\n2019-12-30 00:00:00+00 | 783.2075197029583 | 1030.4505832620227 | 1276.7865808567146\n2020-01-06 00:00:00+00 | 865.2941219994462 | 1096.0356855737048 |  1331.649176312383\n2020-01-13 00:00:00+00 | 834.6747915021757 |  1060.024660266383 |    1286.1810386717\n2020-01-20 00:00:00+00 | 728.2421431793433 |  955.3913494459423 |  1203.730690023456\n2020-01-27 00:00:00+00 | 655.1143367116582 |  903.4836014674186 | 1167.7058289748031\n\n```\n\nIt is also possible to combine the weekly aggregates to run queries on the entire data:\n```SQL,ignore\nSELECT\n    approx_percentile(0.01, combined.digest) AS low,\n    approx_percentile(0.5, combined.digest) AS mid,\n    approx_percentile(0.99, combined.digest) AS high\nFROM (SELECT rollup(digest) AS digest FROM weekly_sketch) AS combined;\n```\n```ignore\n       low        |        mid         |        high\n------------------+--------------------+--------------------\n746.7844638729881 | 1026.6100299252928 | 1294.5391132795592\n```\n\n\n## Command List (A-Z) <a id=\"tdigest-api\"></a>\nAggregate Functions\n> - [tdigest (point form)](#tdigest)\n> - [rollup (summary form)](#tdigest-summary)\n\nAccessor Functions\n> - [approx_percentile](#tdigest_quantile)\n> - [approx_percentile_rank](#tdigest_quantile_at_value)\n> - [max_val](#tdigest_max)\n> - [mean](#tdigest_mean)\n> - [min_val](#tdigest_min)\n> - [num_vals](#tdigest_count)\n\n---\n\n## **tdigest (point form)** <a id=\"tdigest\"></a>\n```SQL ,ignore\ntdigest(\n    buckets INTEGER,\n    value DOUBLE PRECISION\n) RETURNS TDigest\n```\n\nThis will construct and return a TDigest with the specified number of buckets over the given values.\n\n### Required Arguments <a id=\"tdigest-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `buckets` | `INTEGER` | Number of buckets in the digest.  Increasing this will provide more accurate quantile estimates, but will require more memory.|\n| `value` | `DOUBLE PRECISION` |  Column to aggregate.\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `tdigest` | `TDigest` | A t-digest object which may be passed to other t-digest APIs. |\n<br>\n\n### Sample Usages <a id=\"tdigest-examples\"></a>\nFor this example, assume we have a table 'samples' with a column 'weights' holding `DOUBLE PRECISION` values.  The following will simply return a digest over that column\n\n```SQL ,ignore\nSELECT tdigest(100, data) FROM samples;\n```\n\nIt may be more useful to build a view from the aggregate that can later be passed to other tdigest functions.\n\n```SQL ,ignore\nCREATE VIEW digest AS\n    SELECT tdigest(100, data)\n    FROM samples;\n```\n\n---\n\n## **rollup (summary form)** <a id=\"tdigest-summary\"></a>\n```SQL ,ignore\nrollup(\n    digest TDigest\n) RETURNS TDigest\n```\n\nThis will combine multiple already constructed TDigests, if they were created with the same size. This is very useful for re-aggregating digests already constructed using the [point form](#tdigest).  Note that the resulting digest may be subtly different from a digest constructed directly from the underlying points, as noted in the [details section](#tdigest-details) above.\n\n### Required Arguments <a id=\"tdigest-summary-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `digest` | `TDigest` | Previously constructed TDigest objects. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `tdigest` | `TDigest` | A TDigest representing all of the underlying data from all the subaggregates. |\n<br>\n\n### Sample Usages <a id=\"tdigest-summary-examples\"></a>\nThis example assumes a table 'samples' with a column 'data' holding `DOUBLE PRECISION` values and an 'id' column that holds the what series the data belongs to.  A view to get the TDigests for each `id` using the [point form](#tdigest-point) can be created like so:\n\n```SQL ,ignore\nCREATE VIEW digests AS\n    SELECT\n        id,\n        rollup(100, data) as digest\n    FROM samples\n    GROUP BY id;\n```\n\nThat view can then be used to get the full aggregate like so:\n\n```SQL ,ignore\nSELECT rollup(digest)\nFROM digests;\n```\n\n---\n\n## **approx_percentile** <a id=\"tdigest_quantile\"></a>\n\n```SQL ,ignore\napprox_percentile(\n    quantile DOUBLE PRECISION,\n    digest TDigest\n) RETURNS TDigest\n```\n\nGet the approximate value at a quantile from a t-digest\n\n### Required Arguments <a id=\"tdigest_quantile-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `quantile` | `DOUBLE PRECISION` | The desired quantile (0.0-1.0) to approximate. |\n| `digest` | `TDigest` | The digest to compute the quantile on. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `approx_percentile` | `DOUBLE PRECISION` | The estimated value at the requested quantile. |\n<br>\n\n### Sample Usage <a id=\"tdigest_quantile-examples\"></a>\n\n```SQL\nSELECT approx_percentile(0.90, tdigest(100, data))\nFROM generate_series(1, 100) data;\n```\n```output\n approx_percentile\n----------\n     90.5\n```\n\n---\n\n## **approx_percentile_rank** <a id=\"tdigest_quantile_at_value\"></a>\n\n```SQL ,ignore\napprox_percentile_rank(\n    value DOUBLE PRECISION,\n    digest TDigest\n) RETURNS TDigest\n```\n\nEstimate what quantile a given value would be located at in a t-digest.\n\n### Required Arguments <a id=\"tdigest_quantile_at_value-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `value` | `DOUBLE PRECISION` |  The value to estimate the quantile of. |\n| `digest` | `TDigest` | The digest to compute the quantile on. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `approx_percentile_rank` | `DOUBLE PRECISION` | The estimated quantile associated with the provided value. |\n<br>\n\n### Sample Usage <a id=\"tdigest_quantile_at_value-examples\"></a>\n\n```SQL\nSELECT approx_percentile_rank(90, tdigest(100, data))\nFROM generate_series(1, 100) data;\n```\n```output\n approx_percentile_rank\n-------------------\n             0.895\n```\n\n## **max_val** <a id=\"tdigest_max\"></a>\n\n```SQL ,ignore\nmax_val(digest TDigest) RETURNS DOUBLE PRECISION\n```\n\nGet the maximum value from a t-digest.\n\n### Required Arguments <a id=\"tdigest_max-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `digest` | `TDigest` | The digest to extract the max value from. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `max_val` | `DOUBLE PRECISION` | The maximum value entered into the t-digest. |\n<br>\n\n### Sample Usage <a id=\"tdigest_max-examples\"></a>\n\n```SQL\nSELECT max_val(tdigest(100, data))\nFROM generate_series(1, 100) data;\n```\n```output\n max_val\n---------\n     100\n```\n\n---\n\n## **mean** <a id=\"tdigest_mean\"></a>\n\n```SQL ,ignore\nmean(digest TDigest) RETURNS DOUBLE PRECISION\n```\n\nGet the average of all the values contained in a t-digest.\n\n### Required Arguments <a id=\"tdigest_mean-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `digest` | `TDigest` |  The digest to extract the mean value from. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `mean` | `DOUBLE PRECISION` | The average of the values entered into the t-digest. |\n<br>\n\n### Sample Usage <a id=\"tdigest_mean-examples\"></a>\n\n```SQL\nSELECT mean(tdigest(100, data))\nFROM generate_series(1, 100) data;\n```\n```output\n mean\n------\n 50.5\n```\n\n---\n\n## **min_val** <a id=\"tdigest_min\"></a>\n\n```SQL ,ignore\nmin_val(digest TDigest) RETURNS DOUBLE PRECISION\n```\n\nGet the minimum value from a t-digest.\n\n### Required Arguments <a id=\"tdigest_min-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `digest` | `TDigest` | The digest to extract the min value from. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `min_val` | `DOUBLE PRECISION` | The minimum value entered into the t-digest. |\n<br>\n\n### Sample Usages <a id=\"tdigest-min-examples\"></a>\n\n```SQL\nSELECT min_val(tdigest(100, data))\nFROM generate_series(1, 100) data;\n```\n```output\n min_val\n-----------\n         1\n```\n\n---\n\n## **num_vals** <a id=\"tdigest_count\"></a>\n\n```SQL ,ignore\nnum_vals(digest TDigest) RETURNS DOUBLE PRECISION\n```\n\nGet the number of values contained in a t-digest.\n\n### Required Arguments <a id=\"tdigest_count-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `digest` | `TDigest` | The digest to extract the number of values from. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `num_vals` | `DOUBLE PRECISION` | The number of values entered into the t-digest. |\n<br>\n\n### Sample Usage <a id=\"tdigest_count-examples\"></a>\n\n```SQL\nSELECT num_vals(tdigest(100, data))\nFROM generate_series(1, 100) data;\n```\n```output\n num_vals\n-----------\n       100\n```\n\n---\n"
  },
  {
    "path": "docs/template.md",
    "content": "# FEATURE-NAME [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes)\n\n- Current status: ( prototype | experimental | stabilizing | stable )\n- Effort remaining:  ( little | some | lots )\n\nThis is a living document.\n\n## Purpose\n\n- How will this be used?\n- What problem is the user trying to solve?\n- What kind of SQL are they going to write?\n- Is there pure SQL query we are simplifying?\n\n## Use cases\n\n- e.g. single groupings and multiple groupings (not just on `\"time\"`)\n\n### Test Data\n\nExamples below are tested against the following data:\n\n```SQL ,non-transactional\nSET TIME ZONE 'UTC';\nCREATE TABLE example(time TIMESTAMPTZ, value DOUBLE PRECISION);\n```\n\nTODO It would be nice not to have to front-load this.  It shouldn't be too\nhard to mark prereq blocks as such so update-tester can find it and run those\nblocks first.\n\n### simple use case\n\n```SQL\n```\n```output\n```\n\n### complex use cases\n\n### edge cases\n\n## Common functionality\n\nFor aggregates, list our common function overloads here and how this aggregate\nimplements them, or why it doesn't.\n\n### rollup\n\n### into_values / unnest\n\nIs there a need to return a set from the aggregate?\n\n## Implementation plan\n\n### Current status\n\n### Next steps\n\nFirst step is a simple use case in `toolkit_experimental`.\n\nOther steps may include:\n- expanded functionality\n- adjusting based on user feedback\n- optimization\n\nAnd finally:  stabilization or removal.\n\n## Performance (aspirational)\n\nnotes on expectations, current status, future goals\n\nTODO we'll need to document our approach to benchmarking first\ntalk to other groups (who?  query experience?)\n\nFor example if there's a pure SQL way to accomplish a goal and we're just\noffering an improvement, we ought to measure both and show the results.\n\n## Alternatives\n\nBe sure to list alternatives considered and how we chose this approach.\n\n```SQL ,ignore\n[SQL that doesn't work because we didn't implement it]\n```\n"
  },
  {
    "path": "docs/test_caggs.md",
    "content": "# Continuous aggregation tests\n\nThis document serves as a driver for allowing our doctester to verify the behavior of some of our features on continuous aggregates.  It is not intended to serve as documentation, though it does present an example of using continuous aggregates with some toolkit code.\n\nWe're also going to adjust extra_float_digits to use 12 significant digits.  This prevents some spurious failures in the skewness and kurtosis accessors used below.\n```SQL ,non-transactional,ignore-output\nSET extra_float_digits = -3;\n```\n\n## Setup table\n```SQL ,non-transactional,ignore-output\nSET TIME ZONE 'UTC';\nCREATE TABLE test(time TIMESTAMPTZ, value1 DOUBLE PRECISION, value2 DOUBLE PRECISION);\nSELECT create_hypertable('test', 'time');\n```\n\n## Setup continuous aggs\n```SQL ,non-transactional,ignore-output\nCREATE MATERIALIZED VIEW weekly_aggs\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT\n    time_bucket('7 day'::interval, time) as week,\n    hyperloglog(64, value1) as hll,\n    counter_agg(time, value1) as counter,\n    stats_agg(value1, value2) as stats,\n    timevector(time, value2) as tvec,\n    heartbeat_agg(time, time_bucket('7 day'::interval, time), '1w', '55m') as hb\nFROM test\nGROUP BY time_bucket('7 day'::interval, time);\n```\n\n## Populate table\n\n```SQL ,non-transactional,ignore-output\nINSERT INTO test\n    SELECT '2020-01-01'::TIMESTAMPTZ + '1 hour'::INTERVAL * row_number() OVER (),\n        v.b, v.b::DOUBLE PRECISION/v.a::DOUBLE PRECISION\n    FROM (SELECT a, generate_series(a, 100) AS b FROM generate_series(1, 100) a) v;\n```\n\n## Validate continuous aggs\n\n```SQL\nSELECT week, distinct_count(hll), rate(counter), skewness_x(stats, 'population')\nFROM weekly_aggs\nWHERE week > '2020-06-01'::TIMESTAMPTZ\nORDER BY week;\n```\n\n```output\n          week          | distinct_count |       rate        |    skewness_x\n------------------------+----------------+-------------------+-------------------\n 2020-06-08 00:00:00+00 |             49 | 0.000627079174983 |  0.0970167274813\n 2020-06-15 00:00:00+00 |             45 |  0.00065369261477 | -0.0885157388226\n 2020-06-22 00:00:00+00 |             42 | 0.000680306054558 |  0.0864685035294\n 2020-06-29 00:00:00+00 |             36 | 0.000706919494345 | -0.0257336371983\n 2020-07-06 00:00:00+00 |             31 | 0.000971390552229 |   0.169001960922\n 2020-07-13 00:00:00+00 |             28 |   0.0011626746507 |  0.0432068720231\n 2020-07-20 00:00:00+00 |             22 |  0.00168330006653 |   0.344413728361\n 2020-07-27 00:00:00+00 |             10 |  0.00432471264368 |   0.624916113283\n```\n\n```SQL\nSELECT distinct_count(rollup(hll)), stderror(rollup(hll))\nFROM weekly_aggs;\n```\n\n```output\n distinct_count | stderror\n----------------+----------\n            115 |     0.13\n```\n\n```SQL\nSELECT num_resets(rollup(counter))\nFROM weekly_aggs;\n```\n\n```output\n num_resets\n------------\n         98\n```\n\n```SQL\nSELECT average_y(rollup(stats)), stddev_y(rollup(stats)), skewness_y(rollup(stats), 'population'), kurtosis_y(rollup(stats), 'population')\nFROM weekly_aggs;\n```\n\n```output\n average_y |   stddev_y    |   skewness_y    |   kurtosis_y\n-----------+---------------+-----------------+----------------\n        67 | 23.6877840059 | -0.565748443434 | 2.39964349376\n```\n\n```SQL\nSELECT week, count(*)\nFROM (\n    SELECT week, unnest(tvec)\n    FROM weekly_aggs\n    WHERE week > '2020-06-01'::TIMESTAMPTZ\n) s\nGROUP BY week\nORDER BY week;\n```\n```output\n          week          | count \n------------------------+-------\n 2020-06-08 00:00:00+00 |   168\n 2020-06-15 00:00:00+00 |   168\n 2020-06-22 00:00:00+00 |   168\n 2020-06-29 00:00:00+00 |   168\n 2020-07-06 00:00:00+00 |   168\n 2020-07-13 00:00:00+00 |   168\n 2020-07-20 00:00:00+00 |   168\n 2020-07-27 00:00:00+00 |    59\n```\n\n```SQL\nSELECT week, uptime(hb), interpolated_uptime(hb, LAG(hb) OVER (ORDER BY week))\nFROM weekly_aggs\nWHERE week > '2020-06-01'\nORDER BY week;\n```\n```output\n          week          |     uptime      | interpolated_uptime \n------------------------+-----------------+---------------------\n 2020-06-08 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-06-15 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-06-22 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-06-29 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-07-06 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-07-13 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-07-20 00:00:00+00 | 6 days 10:00:00 | 6 days 10:00:00\n 2020-07-27 00:00:00+00 | 2 days 06:05:00 | 2 days 06:05:00\n```\n"
  },
  {
    "path": "docs/test_candlestick_agg.md",
    "content": "# Candlestick Continuous Aggregation Tests\n\n## Setup table\n```SQL,non-transactional,ignore-output\nSET TIME ZONE 'UTC';\nCREATE TABLE stocks_real_time (\n  time TIMESTAMPTZ NOT NULL,\n  symbol TEXT NOT NULL,\n  price DOUBLE PRECISION NULL,\n  day_volume INT NULL\n);\nSELECT create_hypertable('stocks_real_time','time');\n```\n\n## Setup Continuous Aggs\n```SQL,non-transactional,ignore-output\nCREATE MATERIALIZED VIEW cs\nWITH (timescaledb.continuous, timescaledb.materialized_only=false) AS\nSELECT time_bucket('1 minute'::interval, \"time\") AS ts,\n  symbol,\n  candlestick_agg(\"time\", price, day_volume) AS candlestick\nFROM stocks_real_time\nGROUP BY ts, symbol;\n```\n\n## Insert data into table\n```SQL,non-transactional,ignore-output\nINSERT INTO stocks_real_time(\"time\",\"symbol\",\"price\",\"day_volume\")\nVALUES\n('2023-01-11 18:59:59+00','AAPL',140,20),\n('2023-01-11 18:23:58+00','AAPL',100,10),\n('2023-01-11 17:59:57+00','AAPL',133.445,NULL),\n('2023-01-11 17:59:55+00','PFE',47.38,2000),\n('2023-01-11 12:15:55+00','PFE',1,23),\n('2023-01-11 12:00:52+00','AAPL',29.82,NULL),\n('2023-01-11 11:12:12+00','PFE',47.38,14),\n('2023-01-11 11:01:50+00','AMZN',95.25,1000),\n('2023-01-11 11:01:32+00','AMZN',92,NULL),\n('2023-01-11 11:01:30+00','AMZN',75.225,NULL);\n```\n## Query by-minute continuous aggregate over stock trade data for ohlc prices along with timestamps \n\n```SQL,non-transactional\nSELECT ts,\n    symbol,\n    open_time(candlestick),\n    open(candlestick),\n    high_time(candlestick),\n    high(candlestick),\n    low_time(candlestick),\n    low(candlestick),\n    close_time(candlestick),\n    close(candlestick),\n\tvolume(candlestick)\nFROM cs;\n```\n\n```output\n           ts           | symbol |       open_time        |  open   |       high_time        |  high   |        low_time        |   low   |       close_time       |  close  | volume\n------------------------+--------+------------------------+---------+------------------------+---------+------------------------+---------+------------------------+---------+--------\n 2023-01-11 12:15:00+00 | PFE    | 2023-01-11 12:15:55+00 |       1 | 2023-01-11 12:15:55+00 |       1 | 2023-01-11 12:15:55+00 |       1 | 2023-01-11 12:15:55+00 |       1 |     23\n 2023-01-11 17:59:00+00 | PFE    | 2023-01-11 17:59:55+00 |   47.38 | 2023-01-11 17:59:55+00 |   47.38 | 2023-01-11 17:59:55+00 |   47.38 | 2023-01-11 17:59:55+00 |   47.38 |   2000\n 2023-01-11 11:01:00+00 | AMZN   | 2023-01-11 11:01:30+00 |  75.225 | 2023-01-11 11:01:50+00 |   95.25 | 2023-01-11 11:01:30+00 |  75.225 | 2023-01-11 11:01:50+00 |   95.25 |\n 2023-01-11 18:59:00+00 | AAPL   | 2023-01-11 18:59:59+00 |     140 | 2023-01-11 18:59:59+00 |     140 | 2023-01-11 18:59:59+00 |     140 | 2023-01-11 18:59:59+00 |     140 |     20\n 2023-01-11 11:12:00+00 | PFE    | 2023-01-11 11:12:12+00 |   47.38 | 2023-01-11 11:12:12+00 |   47.38 | 2023-01-11 11:12:12+00 |   47.38 | 2023-01-11 11:12:12+00 |   47.38 |     14\n 2023-01-11 17:59:00+00 | AAPL   | 2023-01-11 17:59:57+00 | 133.445 | 2023-01-11 17:59:57+00 | 133.445 | 2023-01-11 17:59:57+00 | 133.445 | 2023-01-11 17:59:57+00 | 133.445 |\n 2023-01-11 18:23:00+00 | AAPL   | 2023-01-11 18:23:58+00 |     100 | 2023-01-11 18:23:58+00 |     100 | 2023-01-11 18:23:58+00 |     100 | 2023-01-11 18:23:58+00 |     100 |     10\n 2023-01-11 12:00:00+00 | AAPL   | 2023-01-11 12:00:52+00 |   29.82 | 2023-01-11 12:00:52+00 |   29.82 | 2023-01-11 12:00:52+00 |   29.82 | 2023-01-11 12:00:52+00 |   29.82 |\n```\n"
  },
  {
    "path": "docs/time_weighted_average.md",
    "content": "# Time Weighted Average\n\n> [Description](#time-weighted-average-description)<br>\n> [Example Usage](time-weighted-average-examples)<br>\n> [API](#time-weighted-average-api) <br>\n> [Notes on Parallelism and Ordering](#time-weight-ordering)<br>\n> [Interpolation Methods Details](#time-weight-methods)<br>\n\n\n## Description <a id=\"time-weighted-average-description\"></a>\n\nTime weighted averages are commonly used in cases where a time series is not evenly sampled, so a traditional average will give misleading results. Consider a voltage sensor that sends readings once every 5 minutes or whenever the value changes by more than 1 V from the previous reading. If the results are generally stable, but with some quick moving transients, a simple average over all of the points will tend to over-weight the transients instead of the stable readings. A time weighted average weights each value by the duration over which it occurred based on the points around it and produces correct results for unevenly spaced series.\n\nTimescaleDB Toolkit's time weighted average is implemented as an aggregate which weights each value either using a last observation carried forward (LOCF) approach or a linear interpolation approach ([see interpolation methods](#time-weight-methods)). While the aggregate is not parallelizable, it is supported with [continuous aggregation](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).\n\nAdditionally, [see the notes on parallelism and ordering](#time-weight-ordering) for a deeper dive into considerations for use with parallelism and some discussion of the internal data structures.\n\n---\n## Example Usage <a id=\"time-weighted-average-examples\"></a>\nFor these examples we'll assume a table `foo` defined as follows, with a bit of example data:\n\n\n```SQL ,non-transactional\nSET TIME ZONE 'UTC';\n CREATE TABLE foo (\n    measure_id      BIGINT,\n    ts              TIMESTAMPTZ ,\n    val             DOUBLE PRECISION,\n    PRIMARY KEY (measure_id, ts)\n);\nINSERT INTO foo VALUES\n( 1, '2020-01-01 00:00:00+00', 10.0),\n( 1, '2020-01-01 00:01:00+00', 20.0),\n( 1, '2020-01-01 00:02:00+00',10.0),\n( 1, '2020-01-01 00:03:00+00', 20.0),\n( 1, '2020-01-01 00:04:00+00', 15.0),\n( 2, '2020-01-01 00:00:00+00', 10.0),\n( 2, '2020-01-01 00:01:00+00', 20.0),\n( 2, '2020-01-01 00:02:00+00',10.0),\n( 2, '2020-01-01 00:03:00+00', 20.0),\n( 2, '2020-01-01 00:04:00+00', 10.0),\n( 2, '2020-01-01 00:08:00+00', 10.0),\n( 2, '2020-01-01 00:10:00+00', 30.0),\n( 2, '2020-01-01 00:10:30+00',10.0),\n( 2, '2020-01-01 00:16:30+00', 35.0),\n( 2, '2020-01-01 00:30:00+00', 60.0);\n```\n```output\nINSERT 0 15\n```\nWhere the measure_id defines a series of related points. A simple use would be to calculate the time weighted average over the whole set of points for each `measure_id`. We'll use the LOCF method for weighting:\n\n```SQL\nSELECT measure_id,\n    average(\n        time_weight('LOCF', ts, val)\n    )\nFROM foo\nGROUP BY measure_id\nORDER BY measure_id;\n```\n```output\n measure_id | average\n------------+---------\n          1 |      15\n          2 |   22.25\n```\n(And of course a where clause can be used to limit the time period we are averaging, the measures we're using etc.).\n\n\nWe can also use the [`time_bucket` function](https://docs.timescale.com/latest/api#time_bucket) to produce a series averages in 15 minute buckets:\n```SQL\nSELECT measure_id,\n    time_bucket('5 min'::interval, ts) as bucket,\n    average(\n        time_weight('LOCF', ts, val)\n    )\nFROM foo\nGROUP BY measure_id, time_bucket('5 min'::interval, ts)\nORDER BY measure_id, time_bucket('5 min'::interval, ts);\n```\n```output\n measure_id |         bucket         | average\n------------+------------------------+---------\n          1 | 2020-01-01 00:00:00+00 |      15\n          2 | 2020-01-01 00:00:00+00 |      15\n          2 | 2020-01-01 00:05:00+00 |\n          2 | 2020-01-01 00:10:00+00 |      30\n          2 | 2020-01-01 00:15:00+00 |\n          2 | 2020-01-01 00:30:00+00 |\n```\nNote that in this case, there are several `time_buckets` that have only a single value, these return `NULL` as the average as we cannot take a time weighted average with only a single point in a bucket and no information about points outside the bucket. In many cases we'll have significantly more data here, but for the example we wanted to keep our data set small.\n\n\nOf course this might be more useful if we make a continuous aggregate out of it. We'll first have to make it a hypertable partitioned on the ts column, with a relatively large chunk_time_interval because the data isn't too high rate:\n\n```SQL ,non-transactional,ignore-output\nSELECT create_hypertable('foo', 'ts', chunk_time_interval=> '15 days'::interval, migrate_data => true);\n```\n\nNow we can make our continuous aggregate:\n\n```SQL ,non-transactional, ignore-output\nCREATE MATERIALIZED VIEW foo_5\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT measure_id,\n    time_bucket('5 min'::interval, ts) as bucket,\n    time_weight('LOCF', ts, val)\nFROM foo\nGROUP BY measure_id, time_bucket('5 min'::interval, ts);\n```\n\n\nNote that here, we just use the `time_weight` function. It's often better to do that and simply run the `average` function when selecting from the view like so:\n```SQL\nSELECT\n    measure_id,\n    bucket,\n    average(time_weight)\nFROM foo_5\nORDER BY measure_id, bucket;\n```\n```output\n measure_id |         bucket         | average\n------------+------------------------+---------\n          1 | 2020-01-01 00:00:00+00 |      15\n          2 | 2020-01-01 00:00:00+00 |      15\n          2 | 2020-01-01 00:05:00+00 |\n          2 | 2020-01-01 00:10:00+00 |      30\n          2 | 2020-01-01 00:15:00+00 |\n          2 | 2020-01-01 00:30:00+00 |\n```\nAnd we get the same results as before. It also allows us to re-aggregate from the continuous aggregate into a larger bucket size quite simply:\n\n```SQL\nSELECT\n    measure_id,\n    time_bucket('1 day'::interval, bucket),\n    average(\n            rollup(time_weight)\n    )\nFROM foo_5\nGROUP BY measure_id, time_bucket('1 day'::interval, bucket)\nORDER BY measure_id, time_bucket('1 day'::interval, bucket);\n```\n```output\n measure_id |      time_bucket       | average\n------------+------------------------+---------\n          1 | 2020-01-01 00:00:00+00 |      15\n          2 | 2020-01-01 00:00:00+00 |   22.25\n```\n\nWe can also use this to speed up our initial calculation where we're only grouping by measure_id and producing a full average (assuming we have a fair number of points per 5 minute period, here it's not going to do much because of our limited example data, but you get the gist):\n\n```SQL\nSELECT\n    measure_id,\n    average(\n        rollup(time_weight)\n    )\nFROM foo_5\nGROUP BY measure_id\nORDER BY measure_id;\n```\n```output\n measure_id | average\n------------+---------\n          1 |      15\n          2 |   22.25\n```\n---\n\n## Command List (A-Z) <a id=\"time-weighted-average-api\"></a>\n> - [time_weight() (point form)](#time_weight_point)\n> - [rollup() (summary form)](#time-weight-summary)\n> - [average()](#time-weight-average)\n\n---\n## **time_weight() (point form)** <a id=\"time_weight_point\"></a>\n```SQL ,ignore\ntime_weight(\n    method TEXT¹,\n    ts TIMESTAMPTZ,\n    value DOUBLE PRECISION\n) RETURNS TimeWeightSummary\n```\n¹ Only two values are currently supported, 'linear' and 'LOCF', any capitalization of these will be accepted. [See interpolation methods for more info.](#time-weight-methods)\n\nAn aggregate that produces a `TimeWeightSummary` from timestamps and associated values.\n\n### Required Arguments² <a id=\"time-weight-point-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `method` | `TEXT` | The weighting method we should use, options are 'linear' or 'LOCF', not case sensitive |\n| `ts` | `TIMESTAMPTZ` |  The time at each point |\n| `value` | `DOUBLE PRECISION` | The value at each point to use for the time weighted average|\n<br>\n\n##### ² Note that `ts` and `value` can be `null`, however the aggregate is not evaluated on `null` values and will return `null`, but it will not error on `null` inputs.\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `time_weight` | `TimeWeightSummary` | A TimeWeightSummary object that can be passed to other functions within the time weighting API. |\n<br>\n\n### Sample Usage\n```SQL ,ignore-output\nWITH t as (\n    SELECT\n        time_bucket('1 day'::interval, ts) as dt,\n        time_weight('Linear', ts, val) AS tw -- get a time weight summary\n    FROM foo\n    WHERE measure_id = 10\n    GROUP BY time_bucket('1 day'::interval, ts)\n)\nSELECT\n    dt,\n    average(tw) -- extract the average from the time weight summary\nFROM t;\n```\n\n## **rollup() (summary form)** <a id=\"time-weight-summary\"></a>\n```SQL ,ignore\nrollup(\n    tws TimeWeightSummary\n) RETURNS TimeWeightSummary\n```\n\nAn aggregate to compute a combined `TimeWeightSummary` from a series of non-overlapping `TimeWeightSummaries`. Non-disjoint `TimeWeightSummaries` will cause errors. See [Notes on Parallelism and Ordering](#time-weight-ordering) for more information.\n\n### Required Arguments² <a id=\"time-weight-summary-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `tws` | `TimeWeightSummary` | The input TimeWeightSummary from a previous `time_weight` (point form) call, often from a [continuous aggregate](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates)|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `time_weight` | `TimeWeightSummary` | A TimeWeightSummary object that can be passed to other functions within the time weighting API. |\n<br>\n\n### Sample Usage\n```SQL ,ignore-output\nWITH t as (\n    SELECT\n        date_trunc('day', ts) as dt,\n        time_weight('Linear', ts, val) AS tw -- get a time weight summary\n    FROM foo\n    WHERE measure_id = 10\n    GROUP BY date_trunc('day', ts)\n), q as (\n    SELECT rollup(tw) AS full_tw -- do a second level of aggregation to get the full time weighted average\n    FROM t\n)\nSELECT\n    dt,\n    average(tw),  -- extract the average from the time weight summary\n    average(tw) / (SELECT average(full_tw) FROM q LIMIT 1)  as normalized -- get the normalized average\nFROM t;\n```\n\n## **average()** <a id=\"time-weight-average\"></a>\n```SQL ,ignore\naverage(\n    tws TimeWeightSummary\n) RETURNS DOUBLE PRECISION\n```\n\nA function to compute a time weighted average from a `TimeWeightSummary`.\n\n### Required Arguments <a id=\"time-weight-summary-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `tws` | `TimeWeightSummary` | The input TimeWeightSummary from a `time_weight` call.|\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `average` | `DOUBLE PRECISION` | The time weighted average computed from the `TimeWeightSummary`|\n<br>\n\n### Sample Usage\n\n```SQL ,ignore\nSELECT\n    id,\n    average(tws)\nFROM (\n    SELECT\n        id,\n        time_weight('LOCF', ts, val) AS tws\n    FROM foo\n    GROUP BY id\n) t\n```\n---\n## Notes on Parallelism and Ordering <a id=\"time-weight-ordering\"></a>\n\nThe time weighted average calculations we perform require a strict ordering of inputs and therefore the calculations are not parallelizable in the strict Postgres sense. This is because when Postgres does parallelism it hands out rows randomly, basically as it sees them to workers. However, if your parallelism can guarantee disjoint (in time) sets of rows, the algorithm can be parallelized, just so long as within some time range, all rows go to the same worker. This is the case for both [continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates) and for [distributed hypertables](https://docs.timescale.com/latest/using-timescaledb/distributed-hypertables) (as long as the partitioning keys are in the group by, though the aggregate itself doesn't horribly make sense otherwise).\n\nWe throw an error if there is an attempt to combine overlapping `TimeWeightSummaries`, for instance, in our example above, if you were to try to combine summaries across `measure_id`s it would error. This is because the interpolation techniques really only make sense within a given time series determined by a single `measure_id`. However, given that the time weighted average produced is a dimensionless quantity, a simple average of time weighted average should better represent the variation across devices, so the recommendation for things like baselines across many timevector would be something like:\n\n```SQL ,ignore-output\nWITH t as (SELECT measure_id,\n        average(\n            time_weight('LOCF', ts, val)\n        ) as time_weighted_average\n    FROM foo\n    GROUP BY measure_id)\nSELECT avg(time_weighted_average) -- use the normal avg function to average our time weighted averages\nFROM t;\n```\n\nInternally, the first and last points seen as well as the calculated weighted sum are stored in each `TimeWeightSummary` and used to combine with a neighboring `TimeWeightSummary` when re-aggregation or the Postgres `combine function` is called. In general, the functions support [partial aggregation](https://www.postgresql.org/docs/current/xaggr.html#XAGGR-PARTIAL-AGGREGATES) and partitionwise aggregation in the multinode context, but are not parallelizable (in the Postgres sense, which requires them to accept potentially overlapping input).\n\nBecause they require ordered sets, the aggregates build up a buffer of input data, sort it and then perform the proper aggregation steps. In cases where memory is proving to be too small to build up a buffer of points causing OOMs or other issues, a multi-level aggregate can be useful. Following our example from above:\n\n```SQL ,ignore-output\nWITH t as (SELECT measure_id,\n    time_bucket('1 day'::interval, ts),\n    time_weight('LOCF', ts, val)\n    FROM foo\n    GROUP BY measure_id, time_bucket('1 day'::interval, ts)\n    )\nSELECT measure_id,\n    average(\n        rollup(time_weight)\n    )\nFROM t\nGROUP BY measure_id;\n```\n\n\nMoving aggregate mode is not supported by `time_weight` and its use as a window function may be quite inefficient, but it is possible to do so as in:\n\n```SQL ,ignore-output\n\nSELECT measure_id,\n    average(\n        time_weight('LOCF', ts, val) OVER (PARTITION BY measure_id  ORDER BY ts RANGE '15 minutes'::interval PRECEDING)\n    )\nFROM foo;\n```\nWhich will give you the 15 minute rolling time weighted average for each point.\n\n---\n## Interpolation Methods Details <a id=\"time-weight-methods\"></a>\n\nDiscrete time values don't always allow for an obvious calculation of the time weighted average. In order to calculate a time weighted average we need to choose how to weight each value. The two methods we currently use are last observation carried forward (LOCF) and linear interpolation.\n\nIn the LOCF approach, the value is treated as if it remains constant until the next value is seen. The LOCF approach is commonly used when the sensor or measurement device sends measurement only when there is a change in value.\n\nThe linear interpolation approach treats the values between any two measurements as if they lie on the line connecting the two measurements. The linear interpolation approach is used to account for irregularly sampled data where the sensor doesn't provide any guarantees\n\nEssentially, internally, the time weighted average computes a numerical approximation of the integral of the theoretical full time curve based on the discrete sampled points provided. We call this the weighted sum.  For LOCF, the the weighted sum will be equivalent to the area under a stepped curve:\n```\n\n|                        (pt 4)\n|          (pt 2)          *\n|            *-------      |\n|            |       |     |\n|(pt 1)      |       *------\n|  *---------      (pt 3)  |\n|  |                       |\n|__|_______________________|______\n             time\n```\nThe linear interpolation is similar, except here it is more of a sawtooth curve. (And the points are different due to the limitations of the slopes of lines one can \"draw\" using ASCII art).\n```\n|                      (pt 4)\n|                        *\n|           (pt 2)     / |\n|            *       /   |\n|          /   \\   /     |\n|(pt 1)  /       *       |\n|      *      (pt 3)     |\n|      |                 |\n|______|_________________|____________\n             time\n```\n\nHere this ends up being equal to the rectangle with width equal to the duration between two points and height the midpoint between the two magnitudes. Once we have this weighted sum, we can divide by the total duration to get the time weighted average.\n"
  },
  {
    "path": "docs/timeseries.md",
    "content": "# Timevector\n\n> [Description](#timevector-description)<br>\n> [Timevector Pipelines](#timevector-pipelines)<br>\n> [Example](#timevector-example)<br>\n> [API](#timevector-api)\n\n## Description <a id=\"timevector-description\"></a>\n\nA timevector is an intermediate representation of a particular value over time used by the extension.  It is a space efficient representation used to store the result of analytic functions such as [asap_smooth]((asap.md#asap_smooth)) or [lttb]((lttb.md#lttb)).  Data can also be directly aggregated into a timevector and passed to functions which support this representation.  The [unnest](#timevector_unnest) API can be used to get the data back from a timevector.\n\n## Timevector Pipelines <a id=\"timevector-pipelines\"></a>\n\nIn an attempt to streamline the timevector interface and make them as easy to use as possible, we've provided a custom operator `->` for applying common operations to timevector and chaining such operations together.  This is much more fully documented in the [timevector pipeline elements](timevector_pipeline_elements.md) page.\n\n## Usage Example <a id=\"timevector-example\"></a>\n\nFor this example, let's start with a table containing some random test data.\n\n```SQL ,non-transactional,ignore-output\nSET TIME ZONE 'UTC';\nCREATE TABLE test(time TIMESTAMPTZ, value DOUBLE PRECISION);\n```\n\n```SQL ,non-transactional\nINSERT INTO test\n    SELECT time, value\n    FROM toolkit_experimental.generate_periodic_normal_series('2020-01-01 UTC'::timestamptz, rng_seed => 11111);\n```\n```output\nINSERT 0 4032\n```\n\nNow lets capture this data into a time series which we'll store in a view.\n\n```SQL ,non-transactional,ignore-output\nCREATE VIEW series AS SELECT timevector(time, value) FROM test;\n```\n\nWe can now use this timevector to efficiently move the data around to other functions.\n\n```SQL\nSELECT time, value::numeric(10,2) FROM\nunnest((SELECT lttb(timevector, 20) FROM series));\n```\n```output\n          time          |       value\n------------------------+--------------------\n2020-01-01 00:00:00+00 | 1038.44\n2020-01-02 04:20:00+00 | 1325.44\n2020-01-03 14:00:00+00 |  708.82\n2020-01-04 18:30:00+00 | 1328.28\n2020-01-05 16:40:00+00 |  802.20\n2020-01-07 06:00:00+00 | 1298.02\n2020-01-09 11:20:00+00 |  741.08\n2020-01-10 18:40:00+00 | 1357.05\n2020-01-13 08:30:00+00 |  780.32\n2020-01-14 03:40:00+00 | 1408.34\n2020-01-15 01:50:00+00 |  895.15\n2020-01-16 20:30:00+00 | 1335.22\n2020-01-18 07:20:00+00 |  823.08\n2020-01-19 18:10:00+00 | 1245.79\n2020-01-21 10:00:00+00 |  666.48\n2020-01-22 23:10:00+00 | 1182.87\n2020-01-24 09:00:00+00 |  736.47\n2020-01-26 05:20:00+00 | 1197.26\n2020-01-28 08:10:00+00 |  659.63\n2020-01-28 23:50:00+00 |  956.29\n```\n\n\n## Command List (A-Z) <a id=\"timevector-api\"></a>\nAggregate Functions\n> - [timevector (point form)](#timevector)\n> - [rollup (summary form)](#timevector-summary)\n\nAccessor Functions\n> - [unnest](#timevector_unnest)\n\n\n---\n\n## **timevector (point form)** <a id=\"timevector\"></a>\n```SQL ,ignore\ntimevector(\n    time TIMESTAMPTZ,\n    value DOUBLE PRECISION\n) RETURNS Timevector\n```\n\nThis will construct and return timevector object containing the passed in time, value pairs.\n\n### Required Arguments <a id=\"timevector-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `time` | `TIMESTAMPTZ` | Time column to aggregate. |\n| `value` | `DOUBLE PRECISION` | Value column to aggregate. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `timevector` | `Timevector` | A timevector object which can be efficiently used by any of our timevector operations. |\n<br>\n\n### Sample Usages <a id=\"timevector-examples\"></a>\nFor this example, assume we have a table 'samples' with two columns, 'time' and 'weight'.  The following will return that table as a timevector.\n\n```SQL ,ignore\nSELECT timevector(time, weight) FROM samples;\n```\n\n---\n\n## **rollup (summary form)** <a id=\"timevector-summary\"></a>\n```SQL ,ignore\nrollup(\n    series timevector\n) RETURNS timevector\n```\n\nThis will combine multiple already constructed timevectors. This is very useful for re-aggregating series already constructed using the [point form](#timevector).\n\n### Required Arguments <a id=\"timevector-summary-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `series` | `timevector` | Previously constructed timevector objects. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `timevector` | `timevector` | A timevector combining all the underlying series. |\n<br>\n\n### Sample Usages <a id=\"timevector-summary-examples\"></a>\nThis example assumes a table 'samples' with columns 'time', 'data', and 'batch'.  We can create a view containing timevector for each batch like so:\n\n```SQL ,ignore\nCREATE VIEW series AS\n    SELECT\n        batch,\n        timevector(time, data) as batch_series\n    FROM samples\n    GROUP BY batch;\n```\n\nIf we want to operate over the combination of all batches, we can get the timevector for this as follows:\n\n```SQL ,ignore\nSELECT rollup(batch_series)\nFROM series;\n```\n\n---\n\n## **unnest** <a id=\"timevector_unnest\"></a>\n\n```SQL ,ignore\nunnest(\n    series timevector\n) RETURNS TABLE(\"time\" timestamp with time zone, value double precision)\n```\n\nThe unnest function is used to get the (time, value) pairs back out of a timevector object.\n\n### Required Arguments <a id=\"timevector_unnest-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `series` | `timevector` | The series to return the data from. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `unnest` | `TABLE` | The (time,value) records contained in the timevector. |\n<br>\n\n### Sample Usage <a id=\"timevector_unnest-examples\"></a>\n\n```SQL\nSELECT unnest(\n    (SELECT timevector(a.time, a.value)\n    FROM\n        (SELECT time, value\n        FROM toolkit_experimental.generate_periodic_normal_series('2020-01-01 UTC'::timestamptz, 45654))\n        a)\n    )\nLIMIT 10;\n```\n```output\n                 unnest\n-----------------------------------------------\n (\"2020-01-01 00:00:00+00\",1009.8399687963981)\n (\"2020-01-01 00:10:00+00\",873.6326953620166)\n (\"2020-01-01 00:20:00+00\",1045.8138997857413)\n (\"2020-01-01 00:30:00+00\",1075.472021940188)\n (\"2020-01-01 00:40:00+00\",956.0229773008177)\n (\"2020-01-01 00:50:00+00\",878.215079403259)\n (\"2020-01-01 01:00:00+00\",1067.8120522056508)\n (\"2020-01-01 01:10:00+00\",1102.3464544566375)\n (\"2020-01-01 01:20:00+00\",952.9509636893868)\n (\"2020-01-01 01:30:00+00\",1031.9006507123047)\n```\n"
  },
  {
    "path": "docs/timeseries_pipeline_elements.md",
    "content": "# Timevector Pipelines [<sup><mark>experimental</mark></sup>](/docs/README.md#tag-notes)\n\n> [Description](#timevector-pipeline-description)<br>\n> [Example](#timevector-pipeline-example)<br>\n> [Pipeline Elements](#timevector-pipeline-elements)\n\n## Description <a id=\"timevector-pipeline-description\"></a>\n\nTimescale timevector objects are just a convenient and efficient way of tracking a single value over time and are detailed a bit more [here](timevector.md).  One of our primary goals with timevector is that they should be easy and efficient to perform basic operations on, and that is where pipelines enter the picture.  At its simplest, a pipeline is just a timevector connected to a [pipeline element](#timevector-pipeline-elements) via the pipeline operator `->`.  However, most pipeline operations output new timevector, so it's possible to chain many pipeline elements together such that the output from one element become the input to the next.\n\n### A note on operator associativity and grouping\n\nDue to limitations in the PostgresQL parser, custom operators are required to be left associative.  The following pipeline will always result in `elementA` being applied to `timevector` and then `elementB` being applied to the result.\n\n```SQL ,ignore\nSELECT timevector -> elementA -> elementB;\n```\n\nHowever, it is possible to explicitly group elements using parentheses:\n\n```SQL ,ignore\nSELECT timevector -> (elementA -> elementB);\n```\n\nThis will result in a pipeline object being created from elements A and B, which will then be applied to the timevector.  While we don't presently take maximum advantage of this internally, these multiple element pipelines should enable optimizations moving forward.  Therefore, this second form should be preferred where possible.\n\n## Usage Example <a id=\"timevector-pipeline-example\"></a>\n\nFor this example let start with a table of temperatures collected from different devices at different times.\n\n```SQL ,non-transactional,ignore-output\nSET TIME ZONE 'UTC';\nCREATE TABLE test_data(time TIMESTAMPTZ, device INTEGER, temperature DOUBLE PRECISION);\n```\n\nIn order to have some nominally interesting data to look at, let's populate this table with random data covering 30 days of readings over 10 devices.\n\n```SQL ,non-transactional,ignore-output\nINSERT INTO test_data\n    SELECT\n        '2020-01-01 00:00:00+00'::timestamptz + ((test_random() * 2592000)::int * '1 second'::interval),\n        floor(test_random() * 10 + 1),\n        50 + test_random() * 20\n    FROM generate_series(1,10000);\n```\n\nNow suppose we want to know how much the temperature fluctuates on a daily basis for each device.  Using timevector and pipelines can simplify the process of finding the answer:\n```SQL ,non-transactional,ignore-output\nCREATE VIEW daily_delta AS\n    SELECT device,\n        timevector(time, temperature)\n            -> (toolkit_experimental.sort()\n            ->  delta()) AS deltas\n    FROM test_data\n    GROUP BY device;\n```\n\nThis command creates a timevector from the time and temperature columns (grouped by device), sorts them in increasing time, and computes the deltas between values.  Now we can look at the deltas for a specific device.  Note that the output for this test is inaccurate as we've removed some of the pipeline elements for the moment.\n\n```SQL,ignore-output\nSELECT time, value::numeric(4,2) AS delta FROM unnest((SELECT deltas FROM daily_delta WHERE device = 3));\n```\n```output\n          time          | delta\n------------------------+-------\n 2020-01-02 00:00:00+00 | -0.54\n 2020-01-03 00:00:00+00 |  0.29\n 2020-01-04 00:00:00+00 | -0.25\n 2020-01-05 00:00:00+00 |  0.07\n 2020-01-06 00:00:00+00 |  0.80\n 2020-01-07 00:00:00+00 | -0.27\n 2020-01-08 00:00:00+00 | -2.55\n 2020-01-09 00:00:00+00 |  3.51\n 2020-01-10 00:00:00+00 | -0.78\n 2020-01-11 00:00:00+00 | -0.39\n 2020-01-12 00:00:00+00 |  0.55\n 2020-01-13 00:00:00+00 | -0.87\n 2020-01-14 00:00:00+00 |  1.17\n 2020-01-15 00:00:00+00 | -2.49\n 2020-01-16 00:00:00+00 |  0.10\n 2020-01-17 00:00:00+00 |  1.09\n 2020-01-18 00:00:00+00 | -0.09\n 2020-01-19 00:00:00+00 |  1.14\n 2020-01-20 00:00:00+00 | -1.23\n 2020-01-21 00:00:00+00 | -0.29\n 2020-01-22 00:00:00+00 | -0.37\n 2020-01-23 00:00:00+00 |  1.48\n 2020-01-24 00:00:00+00 | -0.52\n 2020-01-25 00:00:00+00 |  1.34\n 2020-01-26 00:00:00+00 | -0.95\n 2020-01-27 00:00:00+00 | -0.65\n 2020-01-28 00:00:00+00 | -0.42\n 2020-01-29 00:00:00+00 |  1.42\n 2020-01-30 00:00:00+00 | -0.66\n```\n\nOr even run one of our device's deltas through lttb to get a nice graphable set of points:\n```SQL\nSELECT (deltas -> toolkit_experimental.lttb(10))::TEXT FROM daily_delta where device = 7;\n```\n```output\n                                                                            text\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n(version:1,num_points:10,flags:1,internal_padding:(0,0,0),points:[(ts:\"2020-01-01 23:45:36+00\",val:0),(ts:\"2020-01-02 00:28:48+00\",val:0.01999999999999602),(ts:\"2020-01-02 17:45:36+00\",val:0.020000000000003126),(ts:\"2020-01-02 17:45:36+00\",val:0),(ts:\"2020-01-03 03:07:12+00\",val:0.020000000000003126),(ts:\"2020-01-03 20:24:00+00\",val:0.01999999999999602),(ts:\"2020-01-03 20:24:00+00\",val:0),(ts:\"2020-01-04 05:45:36+00\",val:0.020000000000003126),(ts:\"2020-01-04 23:02:24+00\",val:0.020000000000003126),(ts:\"2020-01-04 23:02:24+00\",val:0)],null_val:[0,0])\n```\n\n## Current Pipeline Elements(A-Z) <a id=\"timevector-pipeline-elements\"></a>\n\nAs of the current timescale release, these elements are all [experimental](/docs/README.md#tag-notes).\n\n\n> - [delta](#timevector_pipeline_delta)\n> - [lttb](#timevector_pipeline_lttb)\n> - [sort](#sort)\n\n\n---\n\n## **delta** <a id=\"timevector_pipeline_delta\"></a>\n```SQL ,ignore\ndelta(\n) RETURNS TimevectorPipelineElement\n```\n\nThis element will return a new timevector where each point is the difference between the current and preceding value in the input timevector.  The new series will be one point shorter as it will not have a preceding value to return a delta for the first point.\n\n### Required Arguments <a id=\"timevector_pipeline_delta-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n<br>\n\n### Pipeline Execution Returns <a id=\"timevector_pipeline_delta-returns\"></a>\n\n|Column|Type|Description|\n|---|---|---|\n| `timevector` | `Timevector` | The result of applying this pipeline element will be a new time series where each point contains the difference in values from the prior point in the input timevector. |\n<br>\n\n### Sample Usage <a id=\"timevector_pipeline_delta-examples\"></a>\n```SQL\nSELECT time, value\nFROM unnest(\n    (SELECT timevector('2020-01-01'::timestamptz + step * '1 day'::interval, step * step)\n        -> delta()\n    FROM generate_series(1, 5) step)\n);\n```\n```output\n          time          | value\n------------------------+-------\n 2020-01-03 00:00:00+00 |     3\n 2020-01-04 00:00:00+00 |     5\n 2020-01-05 00:00:00+00 |     7\n 2020-01-06 00:00:00+00 |     9\n```\n\n---\n\n## **lttb** <a id=\"timevector_pipeline_lttb\"></a>\n```SQL ,ignore\nlttb(\n    resolution int,\n) RETURNS TimevectorPipelineElement\n```\n\nThis element will return a [largest triangle three buckets](lttb.md#description) approximation of a given timevector.  Its behavior is the same as the lttb function documented [here](lttb.md#lttb), save that it expects the series to be sorted.\n\n```SQL ,ignore\nSELECT lttb(time, value, 40) FROM data;\n```\nis equivalent to\n```SQL ,ignore\nSELECT timevector(time, value) -> sort() -> lttb() FROM data;\n```\n\n### Required Arguments <a id=\"timevector_pipeline_lttb-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `resolution` | `INTEGER` | Number of points the output should have. |\n<br>\n\n### Pipeline Execution Returns <a id=\"timevector_pipeline_lttb-returns\"></a>\n\n|Column|Type|Description|\n|---|---|---|\n| `timevector` | `Timevector` | The result of applying this pipeline element will be a new timevector with `resolution` point that is visually similar to the input series. |\n<br>\n\n### Sample Usage <a id=\"timevector_pipeline_lttb-examples\"></a>\n```SQL\nSELECT time, value\nFROM unnest(\n    (SELECT timevector('2020-01-01 UTC'::TIMESTAMPTZ + make_interval(days=>(foo*10)::int), 10 + 5 * cos(foo))\n        -> toolkit_experimental.lttb(4)\n    FROM generate_series(1,11,0.1) foo)\n);\n```\n```output\n          time          |       value\n------------------------+--------------------\n 2020-01-11 00:00:00+00 |   12.7015115293407\n 2020-02-01 00:00:00+00 |  5.004324248633603\n 2020-03-03 00:00:00+00 | 14.982710485116087\n 2020-04-20 00:00:00+00 | 10.022128489940254\n```\n\n---\n\n## **sort** <a id=\"timevector_pipeline_sort\"></a>\n```SQL ,ignore\nsort(\n) RETURNS TimevectorPipelineElement\n```\n\nThis element takes in a timevector and returns a timevector consisting of the same points, but in order of increasing time values.\n\n### Required Arguments <a id=\"timevector_pipeline_sort-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n<br>\n\n### Pipeline Execution Returns <a id=\"timevector_pipeline_sort-returns\"></a>\n\n|Column|Type|Description|\n|---|---|---|\n| `timevector` | `Timevector` | The result of applying this pipeline element will be a time sorted version of the incoming timevector. |\n<br>\n\n### Sample Usage <a id=\"timevector_pipeline_sort-examples\"></a>\n```SQL\nSELECT time, value\nFROM unnest(\n    (SELECT timevector('2020-01-06'::timestamptz - step * '1 day'::interval, step * step)\n        -> toolkit_experimental.sort()\n    FROM generate_series(1, 5) step)\n);\n```\n```output\n          time          | value\n------------------------+-------\n 2020-01-01 00:00:00+00 |    25\n 2020-01-02 00:00:00+00 |    16\n 2020-01-03 00:00:00+00 |     9\n 2020-01-04 00:00:00+00 |     4\n 2020-01-05 00:00:00+00 |     1\n```\n\n---\n"
  },
  {
    "path": "docs/two-step_aggregation.md",
    "content": "# Two-Step Aggregation - What It Is and Why We Use It\n\n## What is a Two-Step Aggregate <a id=\"two-step-description\"></a>\nYou may have noticed that many of our aggregate functions have two parts to them; first an aggregation step and then second an accessor. For instance:\n\n```SQL , ignore\nSELECT average(time_weight('LOCF', value)) as time_weighted_average FROM foo;\n-- or\nSELECT approx_percentile(0.5, percentile_agg(value)) as median FROM bar;\n```\n\nIn each case there is an inner aggregate function (`time_weight` / `percentile_agg`) and an outer call to an accessor function (`average` / `approx_percentile`). We use this calling convention in multiple places throughout the TimescaleDB Toolkit project.\n\nThe inner aggregate call creates a machine-readable partial form that can be used for multiple purposes. The two-step calling convention is slightly longer than a hypothetical one-step one where we just called `time_weighted_average('LOCF', value)` or `percentile_agg(0.5, val)` directly (these functions don't exist, don't try to use them).\n\nWhile the one-step calling convention is easier for the simple case, it becomes much more difficult and hard to reason about for slightly more complex use-cases detailed in the next section. We wanted the calling convention to remain consistent and easy to reason about so you can take advantage of the same functions even as you start doing more complicated analyses.  This also to keeps the docs consistent and prevents adding special cases everywhere.\n\n## Why We Use Two-Step Aggregates <a id=\"two-step-philosophy\"></a>\nInterestingly, almost all Postgres aggregates do a version of this [under the hood already](https://www.postgresql.org/docs/current/xaggr.html), where they have an internal state used for aggregation and then a final function that displays the output to the user.\n\nSo why do we make this calling convention explicit?\n\n1) It allows different accessor function calls to use the same internal state and not redo work.\n2) It cleanly distinguishes the parameters that affect the aggregate and those that only affect the accessor.\n3) It makes it explicit how and when aggregates can be re-aggregated or \"stacked\" on themselves with logically consistent results. This also helps them better integrate with [continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).\n4) It allows for better retrospective analysis of downsampled data in [continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).\n\nThat might have been gibberish to some, so let's unpack it a bit.\n\n### Accessor functions with additional parameters <a id=\"philosophy-accessor-funcs\"></a>\nThe way the optimizer works, if you run an aggregate like:\n```SQL , ignore\nSELECT avg(val), sum(val), count(val) FROM foo;\n```\nThe internal state of the `avg` is actually the `sum` and the `count` and it just returns `sum / count` in the final step of the aggregate. The optimizer knows, when these functions are used, that it doesn't need to run separate aggregates for each, it can use the same internal function and extract the results it needs. This is great! It can save a lot of work. The problem comes when we do something like `percentile_agg` where we have multiple `approx_percentiles` ie:\n\n```SQL , ignore\nSELECT\n    approx_percentile(0.1, percentile_agg(val)) as p10,\n    approx_percentile(0.5, percentile_agg(val)) as p50,\n    approx_percentile(0.9, percentile_agg(val)) as p90\nFROM foo;\n```\nBecause the aggregate step is the same for all three of the calls, the optimizer can combine all the calls, or I can do so explicitly:\n\n```SQL , ignore\nWITH pct as (SELECT percentile_agg(val) as approx FROM foo)\nSELECT\n    approx_percentile(0.1, approx) as p10,\n    approx_percentile(0.5, approx) as p50,\n    approx_percentile(0.9, approx) as p90\nFROM pct;\n```\nBut the work done in each case will be the same.\n\nIf we were to use the one-step calling convention, the extra input of the percentile we're trying to extract would completely confuse the optimizer, and it would have to redo all the calculation inside the aggregate for each of the values you wanted to extract.\n\nSo, if it were framed like this:\n```SQL , ignore\n-- NB: THIS IS AN EXAMPLE OF AN API WE DECIDED NOT TO USE, IT DOES NOT WORK\nSELECT\n    approx_percentile(0.1, val) as p10,\n    approx_percentile(0.5, val) as p50,\n    approx_percentile(0.9, val) as p90\nFROM foo;\n```\nthe optimizer would be forced to build up the necessary internal state three times rather than just once.\n\nThis is even more apparent when you want to use multiple accessor functions, which may have different numbers or types of inputs:\n\n```SQL , ignore\nSELECT\n    approx_percentile(0.1, percentile_agg(val)) as p10,\n    approx_percentile(0.5, percentile_agg(val)) as p50,\n    approx_percentile(0.9, percentile_agg(val)) as p90,\n    error(percentile_agg(val)),\n    approx_percentile_rank(10000, percentile_agg(val)) as percentile_at_threshold\nFROM foo;\n```\nThe optimizer can easily optimize away the redundant `percentile_agg(val)` calls, but would have much more trouble in the one-step approach.\n\n### Explicit association of parameters with either the aggregation or access step <a id=\"philosophy-explicit-association\"></a>\nThis leads us to our second benefit of the two-step approach. A number of our accessor functions (both completed and planned) take inputs that don't affect how we aggregate the underlying data, but do affect how we extract data from the computed aggregate. If we combine everything into one function, it makes it less clear which is which.\n\nNow, our `percentile_agg` implementation uses the `uddsketch` algorithm under the hood and has some default values for parameters, namely the number of buckets it stores and the target error, but there are cases where we might want to use the full algorithm with custom parameters like so:\n```SQL , ignore\nSELECT\n    approx_percentile(0.5, uddsketch(1000, 0.001, val)) as median, -- 1000 buckets, 0.001 relative error target\n    approx_percentile(0.9, uddsketch(1000, 0.001, val)) as p90,\n    approx_percentile(0.5, uddsketch(100, 0.01, val)) as less_accurate_median -- modify the terms for the aggregate get a new approximation\nFROM foo;\n```\nHere we can see which parameters are for the `uddsketch` aggregate (the number of buckets and the target error), and which arguments are for`approx_percentile` (the approx_percentile we want to extract). The optimizer will correctly combine the calls for the first two `uddsketch` calls but not for the third. It is also more clear to the user what is going on, and that I can't set my target error at read time, but rather only at calculation time (this is especially helpful for understanding the behavior of [continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates)).\n\nCombining all of these into one function, so we can use the one-step approach, can get unwieldy and unclear very quickly (ie imagine something like `approx_percentile_uddsketch(0.5, 1000, 0.001)`).\n<br>\n### Stacked aggregates and [continuous aggregate](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates) integration <a id=\"philosophy-reagg\"></a>\nAggregates can be divided into two classes: ones that are \"stackable\" in their final form and ones that are not.\nWhat I'm calling stackable aggregates are ones like `sum`, `min`, `max` etc. that can be re-aggregated on themselves at different groupings without losing their meaning, ie:\n\n```SQL , ignore\nSELECT sum(val) FROM foo;\n-- is equivalent to:\nSELECT sum(sum)\nFROM\n    (SELECT id, sum(val)\n    FROM foo\n    GROUP BY id) s\n```\n\nA non-stackable aggregate like `avg` doesn't have this property:\n```SQL , ignore\nSELECT avg(val) FROM foo;\n-- is NOT equivalent to:\nSELECT avg(avg)\nFROM\n    (SELECT id, avg(val)\n    FROM foo\n    GROUP BY id) s;\n```\n\nOr to say it more succinctly: the `sum` of a `sum` is the `sum` but the `avg` of an `avg` is not the `avg`. This is the difference between stackable and non-stackable aggregates.\n\nThis is not to say that the `avg` of an `avg` is not a useful piece of information, it can be in some cases, but it isn't always what you want and it can be difficult to actually get the true value for non-stackable aggregates, for instance, for `avg` we can take the `count` and `sum` and divide the `sum` by the `count`, but for many aggregates this is not so obvious and for something like `percentile_agg` __LINK__ with a one-step aggregate, the user would simply have to re-implement most of the algorithm in SQL in order to get the result they want.\n\nTwo-step aggregates expose the internal, re-aggregateable form to the user so they can much more easily do this work, so we've tried to provide two-step aggregates wherever we can. This is especially useful for working with [continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates), so if I create a continuous aggregate like so:\n\n```SQL , ignore\nCREATE MATERIALIZED VIEW foo_15\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT id,\n    time_bucket('15 min'::interval, ts) as bucket,\n    sum(val),\n    percentile_agg(val)\nFROM foo\nGROUP BY id, time_bucket('15 min'::interval, ts);\n```\n\nAnd I want to do a second level of aggregation, say over a day, I can do it over the resulting aggregate with the `percentile_agg` function:\n```SQL , ignore\nSELECT id, time_bucket('1 day'::interval, bucket) as bucket,\n    sum(sum),\n    approx_percentile(0.5, percentile_agg(percentile_agg)) as median\nFROM foo_15\nGROUP BY id, time_bucket('1 day'::interval, bucket)\n```\n\n\n##### NB: There are some two-step aggregates like `tdigest` __ADD LINK? and expose and other bits...__ when we document that function where two-step aggregation can lead to more error or different results, because the algorithm is not deterministic in its re-aggregation, but we will note that clearly in the documentation when that happens, it is unusual.\n\n### Retrospective analysis over downsampled data <a id=\"philosophy-retro\"></a>\n[Continuous aggregates](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates) (or separate aggregation tables powered by a cron job or [user-defined action]( __LINK__ ) ) aren't just used for speeding up queries, they're also used for [data retention]( __LINK__ ). But this can mean that they are very difficult to modify as your data ages. Unfortunately this is also when you are learning more things about the analysis you want to do on your data. By keeping them in their raw aggregate form, the user has the flexibility to apply different accessors to do retrospective analysis. With a one-step aggregate the user needs to determine, say, which percentiles are important when we create the continuous aggregate, with a two-step aggregate the user can simply determine they're going to want an approximate percentile, and then determine when doing the analysis whether they want the median, the 90th, 95th or 1st percentile. No need to modify the aggregate or try to re-calculate from data that may no longer exist in the system.\n"
  },
  {
    "path": "docs/uddsketch.md",
    "content": "# UddSketch\n\n> [Description](#uddsketch-description)<br>\n> [Details](#uddsketch-details)<br>\n> [Example](#uddsketch-example)<br>\n> [Example in a Continuous Aggregates](#uddsketch-cagg-example)<br>\n> [API](#uddsketch-api)\n\n## Description <a id=\"uddsketch-description\"></a>\n\n[UddSketch](https://arxiv.org/pdf/2004.08604.pdf) is a specialization of the [DDSketch](https://arxiv.org/pdf/1908.10693.pdf) data structure.  It follows the same approach of breaking the data range into a series of logarithmically sized buckets such that it can guarantee a maximum relative error for any percentile estimate as long as it knows which bucket that percentile falls in.\n\nWhere UddSketch differs from DDSketch in its behavior when the number of buckets required by a set of values exceeds some predefined maximum.  In these circumstances DDSketch will maintain it's original error bound, but only for a subset of the range of percentiles.  UddSketch, on the other hand, will combine buckets in such a way that it loosens the error bound, but can still estimate all percentile values.\n\nAs an example, assume both sketches were trying to capture an large set of values to be able to estimate percentiles with 1% relative error but were given too few buckets to do so.  The DDSketch implementation would still guarantee 1% relative error, but may only be able to provides estimates in the range (0.05, 0.95).  The UddSketch implementation however, might end up only able to guarantee 2% relative error, but would still be able to estimate all percentiles at that error.\n\n## Details <a id=\"uddsketch-details\"></a>\n\nTimescale's UddSketch implementation is provided as an aggregate function in PostgreSQL.  It does not support moving-aggregate mode, and is not a ordered-set aggregate.  It currently only works with `DOUBLE PRECISION` types, but we're intending to relax this constraint as needed.  UddSketches are partializable and are good candidates for [continuous aggregation](https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).\n\nIt's also worth noting that attempting to set the relative error too small or large can result in breaking behavior.  For this reason, the error is required to fall into the range [1.0e-12, 1.0).\n\n## Usage Example <a id=\"uddsketch-example\"></a>\n\nFor this example we're going to start with a table containing some NOAA weather data for a few weather stations across the US over the past 20 years.\n\n```SQL ,ignore\n\\d weather;\n```\n```\n                         Table \"public.weather\"\n Column  |            Type             | Collation | Nullable | Default\n---------+-----------------------------+-----------+----------+---------\n station | text                        |           |          |\n name    | text                        |           |          |\n date    | timestamp without time zone |           |          |\n prcp    | double precision            |           |          |\n snow    | double precision            |           |          |\n tavg    | double precision            |           |          |\n tmax    | double precision            |           |          |\n tmin    | double precision            |           |          |\n```\n\nNow let's create some UddSketches for our different stations and verify that they're receiving data.\n\n```SQL ,ignore\nCREATE VIEW daily_rain AS\n    SELECT name, uddsketch(100, 0.005, prcp)\n    FROM weather\n    GROUP BY name;\n\nSELECT\n    name,\n    num_vals(uddsketch),\n    error(uddsketch)\nFROM daily_rain;\n```\n```\n                 name                  | num_vals |               error\n---------------------------------------+-----------+---------------------\n PORTLAND INTERNATIONAL AIRPORT, OR US |      7671 |  0.0199975003624472\n LITCHFIELD PARK, AZ US                |      5904 |               0.005\n NY CITY CENTRAL PARK, NY US           |      7671 | 0.03997901311671962\n MIAMI INTERNATIONAL AIRPORT, FL US    |      7671 | 0.03997901311671962\n(4 rows)\n```\n\nNotice that 100 buckets proved to be insufficient to maintain 0.5% relative error for three of our data sets, but they've automatically adjusted their bucket size to maintain the desired bucket limit.\n\nWe can then check some rainfall percentiles to see how our stations compare.\n```SQL ,ignore\nSELECT\n    name,\n    approx_percentile(0.6, uddsketch)\nFROM daily_rain;\n```\n```\n                 name                  |             approx_percentile\n---------------------------------------+----------------------\n PORTLAND INTERNATIONAL AIRPORT, OR US | 0.009850446542334412\n LITCHFIELD PARK, AZ US                |                    0\n NY CITY CENTRAL PARK, NY US           |                    0\n MIAMI INTERNATIONAL AIRPORT, FL US    |                    0\n(4 rows)\n```\n```SQL ,ignore\nSELECT\n    name,\n    approx_percentile(0.9, uddsketch)\nFROM daily_rain;\n```\n```\n                 name                  |           approx_percentile\n---------------------------------------+--------------------\n PORTLAND INTERNATIONAL AIRPORT, OR US | 0.3072142710699281\n LITCHFIELD PARK, AZ US                |                  0\n NY CITY CENTRAL PARK, NY US           | 0.4672895773464223\n MIAMI INTERNATIONAL AIRPORT, FL US    | 0.5483701300878486\n(4 rows)\n```\n```SQL ,ignore\nSELECT\n    name,\n    approx_percentile( 0.995, uddsketch)\nFROM daily_rain;\n```\n```\n                 name                  |           approx_percentile\n---------------------------------------+--------------------\n PORTLAND INTERNATIONAL AIRPORT, OR US | 1.1969797510556823\n LITCHFIELD PARK, AZ US                | 0.7671946655927083\n NY CITY CENTRAL PARK, NY US           | 2.3145312888530807\n MIAMI INTERNATIONAL AIRPORT, FL US    | 2.9423518191328113\n(4 rows)\n```\n\n## Example Using TimeScale Continuous Aggregates <a id=\"uddsketch-cagg-example\"></a>\nTo have a UddSketch over a PostgresQL table which automatically updates as more data is added, we can make use of continuous aggregates.  First, let us create a simple hypertable:\n\n```SQL ,non-transactional,ignore-output\nSET TIME ZONE 'UTC';\nCREATE TABLE test(time TIMESTAMPTZ, value DOUBLE PRECISION);\nSELECT create_hypertable('test', 'time');\n```\n\nNow we'll create a continuous aggregate which will group all the points for each week into a UddSketch:\n```SQL ,non-transactional,ignore-output\nCREATE MATERIALIZED VIEW weekly_sketch\nWITH (timescaledb.continuous, timescaledb.materialized_only=false)\nAS SELECT\n    time_bucket('7 day'::interval, time) as week,\n    uddsketch(100, 0.005, value) as sketch\nFROM test\nGROUP BY time_bucket('7 day'::interval, time);\n```\n\nNext we'll use one of our utility functions, `generate_periodic_normal_series`, to add some data to the table.  Using default arguments, this function will add 28 days of data points at 10 minute intervals.\n```SQL ,non-transactional\nINSERT INTO test\n    SELECT time, value\n    FROM toolkit_experimental.generate_periodic_normal_series('2020-01-01 UTC'::timestamptz, rng_seed => 12345678);\n```\n```\nINSERT 0 4032\n```\n\n<div hidden>\nRefresh to make sure we're exercising our serialization path.\n```SQL ,non-tranactional,ignore-output\nCALL refresh_continuous_aggregate('weekly_sketch', NULL, NULL);\n```\n</div>\n\nFinally, we can query the aggregate to see various approximate percentiles from different weeks.\n```SQL\nSELECT\n    week,\n    error(sketch),\n    approx_percentile(0.01, sketch) AS low,\n    approx_percentile(0.5, sketch) AS mid,\n    approx_percentile(0.99, sketch) AS high\nFROM weekly_sketch\nORDER BY week;\n```\n```output\n          week          | error |        low        |        mid         |        high\n------------------------+-------+-------------------+--------------------+--------------------\n 2019-12-30 00:00:00+00 | 0.005 | 808.3889305072331 |  1037.994095858188 | 1280.5527834239035\n 2020-01-06 00:00:00+00 | 0.005 | 858.3773394302965 |  1091.213645863754 | 1306.4218833642865\n 2020-01-13 00:00:00+00 | 0.005 | 816.5134423716273 | 1058.9631440308738 | 1293.4226606442442\n 2020-01-20 00:00:00+00 | 0.005 | 731.4599430896668 |   958.188678537264 | 1205.9785918127336\n 2020-01-27 00:00:00+00 | 0.005 | 688.8626877028054 |  911.4568854686239 | 1135.7472981488002\n```\n\nWe can also combine the weekly aggregates to run queries on the entire data:\n```SQL\nSELECT\n    error(a.uddsketch),\n    approx_percentile(0.01, a.uddsketch) AS low,\n    approx_percentile(0.5, a.uddsketch) AS mid,\n    approx_percentile(0.99, a.uddsketch) AS high\nFROM (SELECT rollup(sketch) as uddsketch FROM weekly_sketch) AS a;\n```\n```output\n error |       low        |        mid         |        high\n-------+------------------+--------------------+--------------------\n 0.005 | 753.736403199032 | 1027.6657963969128 | 1280.5527834239035\n```\n\n\n## Command List (A-Z) <a id=\"uddsketch-api\"></a>\nAggregate Functions\n> - [uddsketch - point form](#uddsketch-point)\n> - [uddsketch - summary form](#uddsketch-summary)\n\nAccessor Functions\n> - [approx_percentile](#approx_percentile)\n> - [approx_percentile_rank](#approx_percentile_rank)\n> - [error](#error)\n> - [mean](#mean)\n> - [num_vals](#num-vals)\n\n---\n\n## **uddsketch (point form) ** <a id=\"uddsketch-point\"></a>\n```SQL ,ignore\nuddsketch(\n    size INTEGER,\n    max_error DOUBLE PRECISION,\n    value DOUBLE PRECISION\n) RETURNS UddSketch\n```\n\nThis will construct and return a new UddSketch with at most `size` buckets.  The maximum relative error of the UddSketch will be bounded by `max_error` unless it is impossible to do so while with the bucket bound.  If the sketch has had to combine buckets, the new error can be found with the [uddsketch_error](#error) command.\n\nNote that since the error will be increased automatically (roughly doubling at each step) as the number of buckets is exceeded, it is probably worth erring on the side of too small unless you have a good understanding of exactly what your error should be.\n\n### Required Arguments <a id=\"uddsketch-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `size` | `INTEGER` | Maximum number of buckets in the sketch.  Providing a larger value here will make it more likely that the aggregate will able to maintain the desired error, though will potentially increase the memory usage. |\n| `max_error` | `DOUBLE PRECISION` | This is the starting maximum relative error of the sketch, as a multiple of the actual value.  The true error may exceed this if too few buckets are provided for the data distribution. |\n| `value` | `DOUBLE PRECISION` |  Column to aggregate.\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `uddsketch` | `UddSketch` | A UddSketch object which may be passed to other UddSketch APIs. |\n<br>\n\n### Sample Usages <a id=\"uddsketch-examples\"></a>\nFor this example assume we have a table 'samples' with a column 'data' holding `DOUBLE PRECISION` values.  The following will simply return a sketch over that column\n\n```SQL ,ignore\nSELECT uddsketch(100, 0.01, data) FROM samples;\n```\n\nIt may be more useful to build a view from the aggregate that we can later pass to other uddsketch functions.\n\n```SQL ,ignore\nCREATE VIEW sketch AS\n    SELECT uddsketch(100, 0.01, data)\n    FROM samples;\n```\n\n---\n\n## **rollup (summary form)** <a id=\"uddsketch-summary\"></a>\n```SQL ,ignore\nrollup(\n    sketch uddsketch\n) RETURNS UddSketch\n```\n\nThis will combine multiple already constructed UddSketches, they must have the same size in order to be combined. This is very useful for re-aggregating already constructed uddsketches using the [point form](#uddsketch-point).\n\n### Required Arguments <a id=\"uddsketch-summary-required-arguments\"></a>\n|Name| Type |Description|\n|---|---|---|\n| `sketch` | `UddSketch` | The already constructed uddsketch from a previous [uddsketch() (point form)](#uddsketch-point) call. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `uddsketch` | `UddSketch` | A UddSketch object which may be passed to other UddSketch APIs. |\n<br>\n\n### Sample Usages <a id=\"uddsketch-summary-examples\"></a>\nFor this example assume we have a table 'samples' with a column 'data' holding `DOUBLE PRECISION` values, and an 'id' column that holds the what series the data belongs to, we can create a view to get the UddSketches for each `id` using the [point form](#uddsketch-point) like so:\n\n```SQL ,ignore\nCREATE VIEW sketch AS\n    SELECT\n        id,\n        uddsketch(100, 0.01, data) as sketched\n    FROM samples\n    GROUP BY id;\n```\n\nThen we can use that view to get the full aggregate like so:\n\n```SQL ,ignore\nSELECT rollup(sketched)\nFROM sketch;\n```\n\n---\n\n## **approx_percentile** <a id=\"approx_percentile\"></a>\n\n```SQL ,ignore\napprox_percentile(\n    percentile DOUBLE PRECISION,\n    sketch  uddsketch\n) RETURNS DOUBLE PRECISION\n```\n\nGet the approximate value at a percentile from a UddSketch.\n\n### Required Arguments <a id=\"approx_percentile-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `percentile` | `DOUBLE PRECISION` | The desired percentile (0.0-1.0) to approximate. |\n| `sketch` | `UddSketch` | The sketch to compute the approx_percentile on. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `approx_percentile` | `DOUBLE PRECISION` | The estimated value at the requested percentile. |\n<br>\n\n### Sample Usage <a id=\"approx_percentile-examples\"></a>\n\n```SQL\nSELECT approx_percentile(\n    0.90,\n    uddsketch(100, 0.01, data)\n) FROM generate_series(1, 100) data;\n```\n```output\n           approx_percentile\n--------------------\n  90.93094205022494\n```\n\n---\n\n## **approx_percentile_rank** <a id=\"approx_percentile_rank\"></a>\n\n```SQL ,ignore\napprox_percentile_rank(\n    value DOUBLE PRECISION,\n    sketch UddSketch\n) RETURNS UddSketch\n```\n\nEstimate what percentile a given value would be located at in a UddSketch.\n\n### Required Arguments <a id=\"approx_percentile_rank-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `value` | `DOUBLE PRECISION` |  The value to estimate the percentile of. |\n| `sketch` | `UddSketch` | The sketch to compute the percentile on. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `approx_percentile_rank` | `DOUBLE PRECISION` | The estimated percentile associated with the provided value. |\n<br>\n\n### Sample Usage <a id=\"approx_percentile_rank-examples\"></a>\n\n```SQL\nSELECT approx_percentile_rank(\n    90,\n    uddsketch(100, 0.01, data)\n) FROM generate_series(1, 100) data;\n```\n```output\n approx_percentile_rank\n-------------------\n             0.89\n```\n\n---\n\n## **error** <a id=\"error\"></a>\n\n```SQL ,ignore\nerror(sketch UddSketch) RETURNS DOUBLE PRECISION\n```\n\nThis returns the maximum relative error that a percentile estimate will have (relative to the correct value).  This will initially be the same as the `max_error` used to construct the UddSketch, but if the sketch has needed to combine buckets this function will return the new maximum error.\n\n### Required Arguments <a id=\"error-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `sketch` | `UddSketch` | The sketch to determine the error of. |\n<br>\n\n### Returns\n\n|Column|Type|Description|\n|---|---|---|\n| `error` | `DOUBLE PRECISION` | The maximum relative error of any percentile estimate. |\n<br>\n\n### Sample Usages <a id=\"error-examples\"></a>\n\n```SQL\nSELECT error(\n    uddsketch(100, 0.01, data)\n) FROM generate_series(1, 100) data;\n```\n```output\n error\n-------\n  0.01\n```\n\n---\n\n## **mean** <a id=\"mean\"></a>\n\n```SQL ,ignore\nmean(sketch UddSketch) RETURNS DOUBLE PRECISION\n```\n\nGet the average of all the values contained in a UddSketch.\n\n### Required Arguments <a id=\"mean-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `sketch` | `UddSketch` |  The sketch to extract the mean value from. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `mean` | `DOUBLE PRECISION` | The average of the values entered into the UddSketch. |\n<br>\n\n### Sample Usage <a id=\"mean-examples\"></a>\n\n```SQL\nSELECT mean(\n    uddsketch(100, 0.01, data)\n) FROM generate_series(1, 100) data;\n```\n```output\n mean\n------\n 50.5\n```\n\n---\n\n## **num_vals** <a id=\"num-vals\"></a>\n\n```SQL ,ignore\nnum_vals(sketch UddSketch) RETURNS DOUBLE PRECISION\n```\n\nGet the number of values contained in a UddSketch.\n\n### Required Arguments <a id=\"num-vals-required-arguments\"></a>\n|Name|Type|Description|\n|---|---|---|\n| `sketch` | `UddSketch` | The sketch to extract the number of values from. |\n<br>\n\n### Returns\n|Column|Type|Description|\n|---|---|---|\n| `uddsketch_count` | `DOUBLE PRECISION` | The number of values entered into the UddSketch. |\n<br>\n\n### Sample Usage <a id=\"num-vals-examples\"></a>\n\n```SQL\nSELECT num_vals(\n    uddsketch(100, 0.01, data)\n) FROM generate_series(1, 100) data;\n```\n```output\n num_vals\n-----------\n       100\n```\n\n---\n"
  },
  {
    "path": "extension/.gitignore",
    "content": ".DS_Store\n.idea/\n.vscode/\n/target\n*.iml\n**/*.rs.bk\nsql/*.generated.sql\n"
  },
  {
    "path": "extension/Cargo.toml",
    "content": "[package]\nname = \"timescaledb_toolkit\"\nversion = \"1.22.0-dev\"\nedition = \"2021\"\n\n[[bin]]\nname = \"pgrx_embed_timescaledb_toolkit\"\npath = \"./src/bin/pgrx_embed.rs\"\n\n[lib]\ncrate-type = [\"cdylib\", \"lib\"]\n\n[features]\ndefault = [\"pg18\"]\npg15 = [\"pgrx/pg15\", \"pgrx-tests/pg15\"]\npg16 = [\"pgrx/pg16\", \"pgrx-tests/pg16\"]\npg17 = [\"pgrx/pg17\", \"pgrx-tests/pg17\"]\npg18 = [\"pgrx/pg18\", \"pgrx-tests/pg18\"]\npg_test = [\"approx\"]\n\n[dependencies]\n# Keep synchronized with `cargo install --version N.N.N cargo-pgrx` in Readme.md and docker/ci/Dockerfile\n# Also `pgrx-tests` down below in `dev-dependencies`.\npgrx = \"=0.16.1\"\npgrx-macros = \"=0.16.1\"\npgrx-sql-entity-graph = \"=0.16.1\"\nencodings = {path=\"../crates/encodings\"}\nflat_serialize = {path=\"../crates/flat_serialize/flat_serialize\"}\nflat_serialize_macro = {path=\"../crates/flat_serialize/flat_serialize_macro\"}\ntdigest = {path=\"../crates/t-digest\"}\nhyperloglogplusplus = {path=\"../crates/hyperloglogplusplus\"}\nuddsketch = {path=\"../crates/udd-sketch\"}\ncounter-agg = {path=\"../crates/counter-agg\"}\nstats_agg = {path=\"../crates/stats-agg\"}\ntime_weighted_average = {path=\"../crates/time-weighted-average\"}\ntspoint = {path=\"../crates/tspoint\"}\nasap = {path=\"../crates/asap\"}\ncountminsketch = {path=\"../crates/count-min-sketch\"}\n\naggregate_builder = {path=\"../crates/aggregate_builder\"}\n\napprox = {version = \"0.4.0\", optional = true}\nbincode = \"1.3.1\"\nserde = { version = \"1.0\", features = [\"derive\"] }\nonce_cell = \"1.8.0\"\nordered-float = {version = \"1.0\", features = [\"serde\"] }\npaste = \"1.0\"\nrand = { version = \"0.8.3\", features = [\"getrandom\", \"small_rng\"] }\nrand_distr = \"0.4.0\"\nrand_chacha = \"0.3.0\"\nron=\"0.6.0\"\ntera = { version = \"1.17.0\", default-features = false }\ntwofloat = { version = \"0.6.0\", features = [\"serde\"] }\nnum-traits = \"0.2.15\"\n\npest = \"=2.3.0\"\npest_derive = \"=2.3.0\"\n\nspfunc = \"0.1.0\"\nstatrs = \"0.15.0\"\n\n[dev-dependencies]\npgrx-tests = \"=0.16.1\"\napprox = \"0.4.0\"\n"
  },
  {
    "path": "extension/src/accessors/tests.rs",
    "content": "use pgrx::*;\n\nuse super::accessor;\n\n//use crate::{accessor, build};\n\n// TODO don't require that trailing comma\naccessor! { one_field(value: f64,) }\naccessor! { two_fields(a: f64, b: i64,) }\n\n#[test]\nfn one_field_works() {\n    let d: AccessorOneField = accessor_one_field(1.0);\n    assert_eq!(1.0, d.value);\n}\n\n#[test]\nfn two_field_works() {\n    let d: AccessorTwoFields = accessor_two_fields(1.0, 2);\n    assert_eq!((1.0, 2), (d.a, d.b));\n}\n"
  },
  {
    "path": "extension/src/accessors.rs",
    "content": "use pgrx::*;\n\nuse counter_agg::range::I64Range;\n\nuse crate::{build, flatten, pg_type, ron_inout_funcs};\n\nmacro_rules! accessor {\n    (\n        $name: ident (\n            $($field:ident : $typ: tt),* $(,)?\n        )\n    ) => {\n        ::paste::paste! {\n            $crate::pg_type!{\n                #[derive(Debug)]\n                struct [<Accessor $name:camel>] {\n                $($field: $typ,)*\n                }\n            }\n            $crate::ron_inout_funcs!([<Accessor $name:camel>]);\n        }\n        accessor_fn_impl! { $name( $( $field: $typ),* ) }\n    };\n}\n\nmacro_rules! accessor_fn_impl {\n    (\n        $name: ident (\n            $( $field:ident : $typ: tt ),*\n                $(,)?\n        )\n    ) => {\n        ::paste::paste!{\n            #[pg_extern(immutable, parallel_safe, name = \"\" $name \"\")]\n            fn [<accessor_ $name >](\n                $( $field: $typ ),*\n            ) -> [<Accessor $name:camel>] {\n                $crate::build! {\n                    [<Accessor $name:camel>] {\n                        $( $field ),*\n                    }\n                }\n            }\n        }\n    };\n}\n\naccessor! { approx_percentile(\n    percentile: f64,\n) }\n\naccessor! { approx_percentile_rank(\n    value: f64,\n) }\n\naccessor! { num_vals() }\naccessor! { mean() }\naccessor! { error() }\naccessor! { min_val() }\naccessor! { max_val() }\naccessor! { average() }\naccessor! { average_x() }\naccessor! { average_y() }\naccessor! { sum() }\naccessor! { sum_x() }\naccessor! { sum_y() }\naccessor! { slope() }\naccessor! { corr() }\naccessor! { intercept() }\naccessor! { x_intercept() }\naccessor! { determination_coeff() }\naccessor! { distinct_count() }\naccessor! { stderror() }\naccessor! { delta() }\naccessor! { time_delta() }\naccessor! { rate() }\naccessor! { irate_left() }\naccessor! { irate_right() }\naccessor! { idelta_left() }\naccessor! { idelta_right() }\naccessor! { num_elements() }\naccessor! { num_changes() }\naccessor! { num_resets() }\naccessor! { counter_zero_time() }\naccessor! { first_val() }\naccessor! { last_val() }\naccessor! { first_time() }\naccessor! { last_time() }\naccessor! { open() }\naccessor! { close() }\naccessor! { high() }\naccessor! { low() }\naccessor! { open_time() }\naccessor! { high_time() }\naccessor! { low_time() }\naccessor! { close_time() }\naccessor! { live_ranges() }\naccessor! { dead_ranges() }\naccessor! { uptime() }\naccessor! { downtime() }\naccessor! { into_values() }\naccessor! { into_array() }\naccessor! { into_int_values() }\naccessor! { state_timeline() }\naccessor! { state_int_timeline() }\naccessor! { num_live_ranges() }\naccessor! { num_gaps() }\naccessor! { topn() }\n// The rest are more complex, with String or other challenges.  Leaving alone for now.\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorLiveAt {\n        time: u64,\n    }\n}\n\nron_inout_funcs!(AccessorLiveAt);\n\n#[pg_extern(immutable, parallel_safe, name = \"live_at\")]\npub fn accessor_live_at(ts: crate::raw::TimestampTz) -> AccessorLiveAt {\n    unsafe {\n        flatten! {\n            AccessorLiveAt {\n                time: ts.0.value() as u64,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorStdDev {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorStdDev);\n\n#[pg_extern(immutable, parallel_safe, name = \"stddev\")]\npub fn accessor_stddev(method: default!(&str, \"'sample'\")) -> AccessorStdDev {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorStdDev {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorStdDevX {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorStdDevX);\n\n#[pg_extern(immutable, parallel_safe, name = \"stddev_x\")]\npub fn accessor_stddev_x(method: default!(&str, \"'sample'\")) -> AccessorStdDevX {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorStdDevX {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorStdDevY {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorStdDevY);\n\n#[pg_extern(immutable, parallel_safe, name = \"stddev_y\")]\npub fn accessor_stddev_y(method: default!(&str, \"'sample'\")) -> AccessorStdDevY {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorStdDevY {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorVariance {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorVariance);\n\n#[pg_extern(immutable, parallel_safe, name = \"variance\")]\npub fn accessor_variance(method: default!(&str, \"'sample'\")) -> AccessorVariance {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorVariance {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorVarianceX {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorVarianceX);\n\n#[pg_extern(immutable, parallel_safe, name = \"variance_x\")]\npub fn accessor_variance_x(method: default!(&str, \"'sample'\")) -> AccessorVarianceX {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorVarianceX {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorVarianceY {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorVarianceY);\n\n#[pg_extern(immutable, parallel_safe, name = \"variance_y\")]\npub fn accessor_variance_y(method: default!(&str, \"'sample'\")) -> AccessorVarianceY {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorVarianceY {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorSkewness {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorSkewness);\n\n#[pg_extern(immutable, parallel_safe, name = \"skewness\")]\npub fn accessor_skewness(method: default!(&str, \"'sample'\")) -> AccessorSkewness {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorSkewness {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorSkewnessX {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorSkewnessX);\n\n#[pg_extern(immutable, parallel_safe, name = \"skewness_x\")]\npub fn accessor_skewness_x(method: default!(&str, \"'sample'\")) -> AccessorSkewnessX {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorSkewnessX {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorSkewnessY {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorSkewnessY);\n\n#[pg_extern(immutable, parallel_safe, name = \"skewness_y\")]\npub fn accessor_skewness_y(method: default!(&str, \"'sample'\")) -> AccessorSkewnessY {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorSkewnessY {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorKurtosis {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorKurtosis);\n\n#[pg_extern(immutable, parallel_safe, name = \"kurtosis\")]\npub fn accessor_kurtosis(method: default!(&str, \"'sample'\")) -> AccessorKurtosis {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorKurtosis {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorKurtosisX {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorKurtosisX);\n\n#[pg_extern(immutable, parallel_safe, name = \"kurtosis_x\")]\npub fn accessor_kurtosis_x(method: default!(&str, \"'sample'\")) -> AccessorKurtosisX {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorKurtosisX {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorKurtosisY {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorKurtosisY);\n\n#[pg_extern(immutable, parallel_safe, name = \"kurtosis_y\")]\npub fn accessor_kurtosis_y(method: default!(&str, \"'sample'\")) -> AccessorKurtosisY {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorKurtosisY {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorCovar {\n        method: crate::stats_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorCovar);\n\n#[pg_extern(immutable, parallel_safe, name = \"covariance\")]\npub fn accessor_covar(method: default!(&str, \"'sample'\")) -> AccessorCovar {\n    let method_enum = crate::stats_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorCovar {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorExtrapolatedDelta {\n        method: crate::counter_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorExtrapolatedDelta);\n\n#[pg_extern(immutable, parallel_safe, name = \"extrapolated_delta\")]\npub fn accessor_extrapolated_delta(method: &str) -> AccessorExtrapolatedDelta {\n    let method_enum = crate::counter_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorExtrapolatedDelta {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorExtrapolatedRate {\n        method: crate::counter_agg::Method,\n    }\n}\n\n//FIXME string IO\nron_inout_funcs!(AccessorExtrapolatedRate);\n\n#[pg_extern(immutable, parallel_safe, name = \"extrapolated_rate\")]\npub fn accessor_extrapolated_rate(method: &str) -> AccessorExtrapolatedRate {\n    let method_enum = crate::counter_agg::method_kind(method);\n    unsafe {\n        flatten! {\n            AccessorExtrapolatedRate {\n                method: method_enum,\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorWithBounds {\n        lower: i64,\n        upper: i64,\n        range_null: u8,\n        lower_present: u8,\n        upper_present: u8,\n    }\n}\n\nron_inout_funcs!(AccessorWithBounds);\n\n#[pg_extern(immutable, parallel_safe, name = \"with_bounds\")]\npub fn accessor_with_bounds(bounds: crate::raw::tstzrange) -> AccessorWithBounds {\n    let range = unsafe { crate::range::get_range(bounds.0.cast_mut_ptr()) };\n    let mut accessor = build! {\n        AccessorWithBounds {\n            lower: 0,\n            upper: 0,\n            range_null: 0,\n            lower_present: 0,\n            upper_present: 0,\n        }\n    };\n    match range {\n        None => accessor.range_null = 1,\n        Some(range) => {\n            if let Some(left) = range.left {\n                accessor.lower_present = 1;\n                accessor.lower = left;\n            }\n            if let Some(right) = range.right {\n                accessor.upper_present = 1;\n                accessor.upper = right;\n            }\n        }\n    }\n    accessor\n}\n\nimpl AccessorWithBounds {\n    pub fn bounds(&self) -> Option<I64Range> {\n        if self.range_null != 0 {\n            return None;\n        }\n\n        I64Range {\n            left: (self.lower_present != 0).then(|| self.lower),\n            right: (self.upper_present != 0).then(|| self.upper),\n        }\n        .into()\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorUnnest {\n    }\n}\n\nron_inout_funcs!(AccessorUnnest);\n\n// Note that this should be able to replace the timescale_experimental.unnest function\n// and related object in src/timevector/pipeline/expansion.rs\n#[pg_extern(immutable, parallel_safe, name = \"unnest\")]\npub fn accessor_unnest() -> AccessorUnnest {\n    build! {\n        AccessorUnnest {\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorIntegral {\n        len: u8,\n        bytes: [u8; 16],\n    }\n}\n\n// FIXME string IO\nron_inout_funcs!(AccessorIntegral);\n\n#[pg_extern(immutable, parallel_safe, name = \"integral\")]\npub fn accessor_integral(unit: default!(&str, \"'second'\")) -> AccessorIntegral {\n    if unit.len() > 16 {\n        pgrx::error!(\n            \"Time unit string too long: {} characters (max 16)\",\n            unit.len()\n        );\n    }\n\n    let mut bytes = [0u8; 16];\n    let unit_bytes = unit.as_bytes();\n    bytes[..unit_bytes.len()].copy_from_slice(unit_bytes);\n\n    unsafe {\n        flatten! {\n            AccessorIntegral {\n                len: unit.len() as u8,\n                bytes,\n            }\n        }\n    }\n}\n\n// Note we also have a AccessorTopn which is similar to this but doesn't store the count\npg_type! {\n    #[derive(Debug)]\n    struct AccessorTopNCount {\n        count: i64,\n    }\n}\n\nron_inout_funcs!(AccessorTopNCount);\n\n#[pg_extern(immutable, parallel_safe, name = \"topn\")]\npub fn accessor_topn_count(count: i64) -> AccessorTopNCount {\n    unsafe {\n        flatten! {\n            AccessorTopNCount {\n                count\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorMaxFrequencyInt {\n        value: i64,\n    }\n}\n\nron_inout_funcs!(AccessorMaxFrequencyInt);\n\n#[pg_extern(immutable, parallel_safe, name = \"max_frequency\")]\npub fn accessor_max_frequency_int(value: i64) -> AccessorMaxFrequencyInt {\n    unsafe {\n        flatten! {\n            AccessorMaxFrequencyInt {\n                value\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorMinFrequencyInt {\n        value: i64,\n    }\n}\n\nron_inout_funcs!(AccessorMinFrequencyInt);\n\n#[pg_extern(immutable, parallel_safe, name = \"min_frequency\")]\npub fn accessor_min_frequency_int(value: i64) -> AccessorMinFrequencyInt {\n    unsafe {\n        flatten! {\n            AccessorMinFrequencyInt {\n                value\n            }\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct AccessorPercentileArray {\n        len: u64,\n        percentile: [f64; 32],\n    }\n}\n\nron_inout_funcs!(AccessorPercentileArray);\n\n#[pg_extern(immutable, name = \"approx_percentiles\")]\npub fn accessor_percentiles(unit: Vec<f64>) -> AccessorPercentileArray {\n    if unit.len() > 32 {\n        pgrx::error!(\"Too many percentiles: {} (max 32)\", unit.len());\n    }\n\n    let mut percentile = [0.0f64; 32];\n    for (i, &val) in unit.iter().enumerate() {\n        percentile[i] = val;\n    }\n\n    unsafe {\n        flatten! {\n            AccessorPercentileArray {\n                len: unit.len() as u64,\n                percentile,\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "extension/src/aggregate_builder_tests.rs",
    "content": "// Tests for `aggregate_builder::aggregate`. This can't be in the\n// aggregate_builder crate because it requires too much of postgres to actually\n// function\nuse aggregate_builder::aggregate;\n\nuse pgrx::*;\n\nuse crate::{palloc::Inner, raw::bytea};\n\n// just about the simplest aggregate `arbitrary()` returns an arbitrary element\n// from the input set. We have three versions\n//  1. `anything()` tests that the minimal functionality works.\n//  2. `cagg_anything()` tests that the config we use for caggs (serialization\n//     but not parallel-safe) outputs the expected config.\n//  3. `parallel_anything()` tests that the parallel version outputs the expected\n//      config.\n#[aggregate]\nimpl toolkit_experimental::anything {\n    type State = String;\n\n    fn transition(state: Option<State>, #[sql_type(\"text\")] value: String) -> Option<State> {\n        state.or(Some(value))\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<String> {\n        state.as_deref().cloned()\n    }\n}\n\n#[aggregate]\nimpl toolkit_experimental::cagg_anything {\n    type State = String;\n\n    fn transition(state: Option<State>, #[sql_type(\"text\")] value: String) -> Option<State> {\n        state.or(Some(value))\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<String> {\n        state.as_deref().cloned()\n    }\n\n    fn serialize(state: &State) -> bytea {\n        crate::do_serialize!(state)\n    }\n\n    fn deserialize(bytes: bytea) -> State {\n        crate::do_deserialize!(bytes, State)\n    }\n\n    fn combine(a: Option<&State>, b: Option<&State>) -> Option<State> {\n        a.or(b).cloned()\n    }\n}\n\n#[aggregate]\nimpl toolkit_experimental::parallel_anything {\n    type State = String;\n\n    fn transition(state: Option<State>, #[sql_type(\"text\")] value: String) -> Option<State> {\n        state.or(Some(value))\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<String> {\n        state.as_deref().cloned()\n    }\n\n    const PARALLEL_SAFE: bool = true;\n\n    fn serialize(state: &State) -> bytea {\n        crate::do_serialize!(state)\n    }\n\n    fn deserialize(bytes: bytea) -> State {\n        crate::do_deserialize!(bytes, State)\n    }\n\n    fn combine(a: Option<&State>, b: Option<&State>) -> Option<State> {\n        a.or(b).cloned()\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_anything_in_experimental_and_returns_first() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\n                    \"SELECT toolkit_experimental.anything(val) \\\n                FROM (VALUES ('foo'), ('bar'), ('baz')) as v(val)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(output.as_deref(), Some(\"foo\"));\n        })\n    }\n\n    #[pg_test]\n    fn test_anything_has_correct_fn_names_and_def() {\n        Spi::connect_mut(|client| {\n            let spec = get_aggregate_spec(client, \"anything\");\n            // output is\n            //   fn kind (`a`), volatility, parallel-safety, num args, final fn modify (is this right?)\n            //   transition type (`internal`)\n            //   output type\n            //   transition fn name,\n            //   final fn name,\n            //   serialize fn name or - if none,\n            //   deserialize fn name or - if none,\n            assert_eq!(\n                spec,\n                \"(\\\n                    a,i,u,1,r,\\\n                    internal,\\\n                    text,\\\n                    toolkit_experimental.anything_transition_fn_outer,\\\n                    toolkit_experimental.anything_finally_fn_outer,\\\n                    -,\\\n                    -,\\\n                    -\\\n                )\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_cagg_anything_has_correct_fn_names_and_def() {\n        Spi::connect_mut(|client| {\n            let spec = get_aggregate_spec(client, \"cagg_anything\");\n            // output is\n            //   fn kind (`a`), volatility, parallel-safety, num args, final fn modify (is this right?)\n            //   transition type (`internal`)\n            //   output type\n            //   transition fn name,\n            //   final fn name,\n            //   serialize fn name or - if none,\n            //   deserialize fn name or - if none,\n            assert_eq!(\n                spec,\n                \"(\\\n                    a,i,u,1,r,\\\n                    internal,\\\n                    text,\\\n                    toolkit_experimental.cagg_anything_transition_fn_outer,\\\n                    toolkit_experimental.cagg_anything_finally_fn_outer,\\\n                    toolkit_experimental.cagg_anything_serialize_fn_outer,\\\n                    toolkit_experimental.cagg_anything_deserialize_fn_outer,\\\n                    toolkit_experimental.cagg_anything_combine_fn_outer\\\n                )\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_parallel_anything_has_correct_fn_names_and_def() {\n        Spi::connect_mut(|client| {\n            let spec = get_aggregate_spec(client, \"parallel_anything\");\n            // output is\n            //   fn kind (`a`), volatility, parallel-safety, num args, final fn modify (is this right?)\n            //   transition type (`internal`)\n            //   output type\n            //   transition fn name,\n            //   final fn name,\n            //   serialize fn name or - if none,\n            //   deserialize fn name or - if none,\n            assert_eq!(\n                spec,\n                \"(\\\n                    a,i,s,1,r,\\\n                    internal,\\\n                    text,\\\n                    toolkit_experimental.parallel_anything_transition_fn_outer,\\\n                    toolkit_experimental.parallel_anything_finally_fn_outer,\\\n                    toolkit_experimental.parallel_anything_serialize_fn_outer,\\\n                    toolkit_experimental.parallel_anything_deserialize_fn_outer,\\\n                    toolkit_experimental.parallel_anything_combine_fn_outer\\\n                )\"\n            );\n        });\n    }\n\n    // It gets annoying, and segfaulty to handle many arguments from the Spi.\n    // For simplicity, we just return a single string representing the tuple\n    // and use string-comparison.\n    fn get_aggregate_spec(client: &mut spi::SpiClient, aggregate_name: &str) -> String {\n        client\n            .update(\n                &format!(\n                    r#\"SELECT (\n                prokind,\n                provolatile,\n                proparallel,\n                pronargs,\n                aggfinalmodify,\n                aggtranstype::regtype,\n                prorettype::regtype,\n                aggtransfn,\n                aggfinalfn,\n                aggserialfn,\n                aggdeserialfn,\n                aggcombinefn)::TEXT\n            FROM pg_proc, pg_aggregate\n            WHERE proname = '{aggregate_name}'\n              AND pg_proc.oid = aggfnoid;\"#\n                ),\n                None,\n                &[],\n            )\n            .unwrap()\n            .first()\n            .get_one::<String>()\n            .unwrap()\n            .expect(\"no aggregate found\")\n    }\n}\n"
  },
  {
    "path": "extension/src/aggregate_utils.rs",
    "content": "use std::ptr::null_mut;\n\nuse pgrx::pg_sys;\n\n// TODO move to func_utils once there are enough function to warrant one\npub unsafe fn get_collation(fcinfo: pg_sys::FunctionCallInfo) -> Option<pg_sys::Oid> {\n    if (*fcinfo).fncollation == pg_sys::Oid::INVALID {\n        None\n    } else {\n        Some((*fcinfo).fncollation)\n    }\n}\n\npub fn get_collation_or_default(fcinfo: pg_sys::FunctionCallInfo) -> Option<pg_sys::Oid> {\n    if fcinfo.is_null() {\n        Some(pg_sys::Oid::from(100)) // TODO: default OID, there should be a constant for this\n    } else {\n        unsafe { get_collation(fcinfo) }\n    }\n}\n\npub unsafe fn in_aggregate_context<T, F: FnOnce() -> T>(\n    fcinfo: pg_sys::FunctionCallInfo,\n    f: F,\n) -> T {\n    let mctx =\n        aggregate_mctx(fcinfo).unwrap_or_else(|| pgrx::error!(\"cannot call as non-aggregate\"));\n    crate::palloc::in_memory_context(mctx, f)\n}\n\npub unsafe fn aggregate_mctx(fcinfo: pg_sys::FunctionCallInfo) -> Option<pg_sys::MemoryContext> {\n    if fcinfo.is_null() {\n        return Some(pg_sys::CurrentMemoryContext);\n    }\n    let mut mctx = null_mut();\n    let is_aggregate = pg_sys::AggCheckCallContext(fcinfo, &mut mctx);\n    if is_aggregate == 0 {\n        None\n    } else {\n        debug_assert!(!mctx.is_null());\n        Some(mctx)\n    }\n}\n"
  },
  {
    "path": "extension/src/asap.rs",
    "content": "use asap::*;\nuse pgrx::*;\nuse serde::{Deserialize, Serialize};\n\nuse crate::{\n    aggregate_utils::in_aggregate_context,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    time_vector,\n};\n\nuse tspoint::TSPoint;\n\nuse crate::time_vector::{Timevector_TSTZ_F64, Timevector_TSTZ_F64Data};\n\n#[derive(Debug, Serialize, Deserialize, Clone)]\npub struct ASAPTransState {\n    ts: Vec<TSPoint>,\n    sorted: bool,\n    resolution: i32,\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn asap_trans(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    resolution: i32,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    asap_trans_internal(unsafe { state.to_inner() }, ts, val, resolution, fcinfo).internal()\n}\npub fn asap_trans_internal(\n    state: Option<Inner<ASAPTransState>>,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    resolution: i32,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<ASAPTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let p = match (ts, val) {\n                (_, None) => return state,\n                (None, _) => return state,\n                (Some(ts), Some(val)) => TSPoint { ts: ts.into(), val },\n            };\n\n            match state {\n                None => Some(\n                    ASAPTransState {\n                        ts: vec![p],\n                        sorted: true,\n                        resolution,\n                    }\n                    .into(),\n                ),\n                Some(mut s) => {\n                    s.add_point(p);\n                    Some(s)\n                }\n            }\n        })\n    }\n}\n\nimpl ASAPTransState {\n    fn add_point(&mut self, point: TSPoint) {\n        self.ts.push(point);\n        if let Some(window) = self.ts.windows(2).last() {\n            if window[0].ts > window[1].ts {\n                self.sorted = false\n            }\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn asap_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    asap_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\nfn asap_final_inner(\n    state: Option<Inner<ASAPTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let state = match state {\n                None => return None,\n                Some(state) => state.clone(),\n            };\n\n            let mut points = state.ts;\n            if !state.sorted {\n                points.sort_by_key(|p| p.ts);\n            }\n\n            let start_ts = points.first().unwrap().ts;\n            let end_ts = points.last().unwrap().ts;\n\n            let mut values: Vec<f64> = points.iter().map(|p| p.val).collect();\n            values = asap_smooth(&values, state.resolution as u32);\n\n            let interval = if values.len() > 1 {\n                (end_ts - start_ts) / (values.len() - 1) as i64\n            } else {\n                1\n            };\n\n            let points: Vec<_> = values\n                .into_iter()\n                .enumerate()\n                .map(|(i, val)| TSPoint {\n                    ts: start_ts + i as i64 * interval,\n                    val,\n                })\n                .collect();\n\n            let nulls_len = points.len().div_ceil(8);\n\n            Some(crate::build! {\n                Timevector_TSTZ_F64 {\n                    num_points: points.len() as u32,\n                    flags: time_vector::FLAG_IS_SORTED,\n                    internal_padding: [0; 3],\n                    points: points.into(),\n                    null_val: std::vec::from_elem(0_u8, nulls_len).into(),\n                }\n            })\n        })\n    }\n}\n\n#[pg_extern(name = \"asap_smooth\", immutable, parallel_safe)]\npub fn asap_on_timevector(\n    mut series: Timevector_TSTZ_F64<'static>,\n    resolution: i32,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    // TODO: implement this using zero copy (requires sort, find_downsample_interval, and downsample_and_gapfill on Timevector)\n    let needs_sort = series.is_sorted();\n\n    if needs_sort {\n        series.points.as_owned().sort_by_key(|p| p.ts);\n    }\n    let start_ts = series.points.as_slice().first().unwrap().ts;\n    let end_ts = series.points.as_slice().last().unwrap().ts;\n\n    let values: Vec<f64> = series.points.as_slice().iter().map(|p| p.val).collect();\n\n    let result = asap_smooth(&values, resolution as u32);\n\n    let interval = if result.len() > 1 {\n        (end_ts - start_ts) / (result.len() - 1) as i64\n    } else {\n        1\n    };\n\n    let points: Vec<_> = result\n        .into_iter()\n        .enumerate()\n        .map(|(i, val)| TSPoint {\n            ts: start_ts + i as i64 * interval,\n            val,\n        })\n        .collect();\n\n    let nulls_len = points.len().div_ceil(8);\n\n    Some(crate::build! {\n        Timevector_TSTZ_F64 {\n            num_points: points.len() as u32,\n            flags: time_vector::FLAG_IS_SORTED,\n            internal_padding: [0; 3],\n            points: points.into(),\n            null_val: std::vec::from_elem(0_u8, nulls_len).into(),\n        }\n    })\n}\n\n// Aggregate on only values (assumes aggregation over ordered normalized timestamp)\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE asap_smooth(ts TIMESTAMPTZ, value DOUBLE PRECISION, resolution INT)\\n\\\n    (\\n\\\n        sfunc = asap_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = asap_final\\n\\\n    );\\n\",\n    name = \"asap_agg\",\n    requires = [asap_trans, asap_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use approx::assert_relative_eq;\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_against_reference() {\n        // Test our ASAP implementation against the reference implementation at http://www.futuredata.io.s3-website-us-west-2.amazonaws.com/asap/\n        // The sample data is the first 100 points of the second sample data set.  Note that the dates are not important for this test.\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            let mut result = client.update(\n                \"\n                SELECT value\n                FROM unnest(\n                (SELECT asap_smooth('2020-1-1'::timestamptz + i * '1d'::interval, val, 10)\n                    FROM (VALUES \n                        (1,1.1),(2,4.4),(3,7.5),(4,8.9),(5,11.7),(6,15),(7,15.3),(8,15.6),(9,13.3),(10,11.1),\n                        (11,7.5),(12,5.8),(13,5.6),(14,4.2),(15,4.7),(16,7.2),(17,11.4),(18,15.3),(19,15),(20,16.2),\n                        (21,14.4),(22,8.6),(23,5.3),(24,3.3),(25,4.4),(26,3.3),(27,5),(28,8.1),(29,10.8),(30,12.2),\n                        (31,13.8),(32,13.3),(33,12.8),(34,9.4),(35,6.9),(36,3.9),(37,1.1),(38,4.2),(39,4.2),(40,8.4),\n                        (41,13.4),(42,16.4),(43,16),(44,15.6),(45,14.7),(46,10.2),(47,6.1),(48,1.8),(49,4.2),(50,5),\n                        (51,5.1),(52,9.2),(53,13.6),(54,14.9),(55,16.9),(56,16.9),(57,14.4),(58,10.8),(59,4.7),(60,3.6),\n                        (61,3.9),(62,2.4),(63,7.1),(64,8.3),(65,12.5),(66,16.4),(67,16.9),(68,16),(69,12.8),(70,9.1),\n                        (71,7.2),(72,1.6),(73,1.2),(74,2.3),(75,2.8),(76,7.1),(77,10.3),(78,15.1),(79,16.8),(80,15.7),\n                        (81,16.6),(82,10.1),(83,8.1),(84,5),(85,4.1),(86,4.7),(87,6.2),(88,8.7),(89,12.4),(90,14),\n                        (91,15.3),(92,16.3),(93,15.3),(94,10.9),(95,9.2),(96,3.4),(97,1.9),(98,2.2),(99,6),(100,6.8)\n                    ) AS v(i, val)\n                )) s\",\n                None,\n            &[]).unwrap();\n\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                10.39\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                9.29\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                7.54\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                7.8\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                10.34\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                11.01\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                10.54\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                8.01\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                8.99\n            );\n            assert_relative_eq!(\n                result.next().unwrap()[1].value::<f64>().unwrap().unwrap() as f32,\n                8.73\n            );\n            assert!(result.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    fn test_asap_equivalence() {\n        Spi::connect_mut(|client| {\n            let mut value_result = client.update(\n                \"\n                SELECT time::text, value\n                FROM unnest(\n                (SELECT asap_smooth('2020-1-1'::timestamptz + i * '1d'::interval, val, 10)\n                    FROM (VALUES \n                        (1,1.1),(2,4.4),(3,7.5),(4,8.9),(5,11.7),(6,15),(7,15.3),(8,15.6),(9,13.3),(10,11.1),\n                        (11,7.5),(12,5.8),(13,5.6),(14,4.2),(15,4.7),(16,7.2),(17,11.4),(18,15.3),(19,15),(20,16.2),\n                        (21,14.4),(22,8.6),(23,5.3),(24,3.3),(25,4.4),(26,3.3),(27,5),(28,8.1),(29,10.8),(30,12.2),\n                        (31,13.8),(32,13.3),(33,12.8),(34,9.4),(35,6.9),(36,3.9),(37,1.1),(38,4.2),(39,4.2),(40,8.4),\n                        (41,13.4),(42,16.4),(43,16),(44,15.6),(45,14.7),(46,10.2),(47,6.1),(48,1.8),(49,4.2),(50,5),\n                        (51,5.1),(52,9.2),(53,13.6),(54,14.9),(55,16.9),(56,16.9),(57,14.4),(58,10.8),(59,4.7),(60,3.6),\n                        (61,3.9),(62,2.4),(63,7.1),(64,8.3),(65,12.5),(66,16.4),(67,16.9),(68,16),(69,12.8),(70,9.1),\n                        (71,7.2),(72,1.6),(73,1.2),(74,2.3),(75,2.8),(76,7.1),(77,10.3),(78,15.1),(79,16.8),(80,15.7),\n                        (81,16.6),(82,10.1),(83,8.1),(84,5),(85,4.1),(86,4.7),(87,6.2),(88,8.7),(89,12.4),(90,14),\n                        (91,15.3),(92,16.3),(93,15.3),(94,10.9),(95,9.2),(96,3.4),(97,1.9),(98,2.2),(99,6),(100,6.8)\n                    ) AS v(i, val)\n                )) s\",\n                None,\n            &[]).unwrap();\n\n            let mut tvec_result = client.update(\n                \"\n                SELECT time::text, value\n                FROM unnest(\n                (SELECT asap_smooth(\n                    (SELECT timevector('2020-1-1'::timestamptz + i * '1d'::interval, val)\n                        FROM (VALUES \n                            (1,1.1),(2,4.4),(3,7.5),(4,8.9),(5,11.7),(6,15),(7,15.3),(8,15.6),(9,13.3),(10,11.1),\n                            (11,7.5),(12,5.8),(13,5.6),(14,4.2),(15,4.7),(16,7.2),(17,11.4),(18,15.3),(19,15),(20,16.2),\n                            (21,14.4),(22,8.6),(23,5.3),(24,3.3),(25,4.4),(26,3.3),(27,5),(28,8.1),(29,10.8),(30,12.2),\n                            (31,13.8),(32,13.3),(33,12.8),(34,9.4),(35,6.9),(36,3.9),(37,1.1),(38,4.2),(39,4.2),(40,8.4),\n                            (41,13.4),(42,16.4),(43,16),(44,15.6),(45,14.7),(46,10.2),(47,6.1),(48,1.8),(49,4.2),(50,5),\n                            (51,5.1),(52,9.2),(53,13.6),(54,14.9),(55,16.9),(56,16.9),(57,14.4),(58,10.8),(59,4.7),(60,3.6),\n                            (61,3.9),(62,2.4),(63,7.1),(64,8.3),(65,12.5),(66,16.4),(67,16.9),(68,16),(69,12.8),(70,9.1),\n                            (71,7.2),(72,1.6),(73,1.2),(74,2.3),(75,2.8),(76,7.1),(77,10.3),(78,15.1),(79,16.8),(80,15.7),\n                            (81,16.6),(82,10.1),(83,8.1),(84,5),(85,4.1),(86,4.7),(87,6.2),(88,8.7),(89,12.4),(90,14),\n                            (91,15.3),(92,16.3),(93,15.3),(94,10.9),(95,9.2),(96,3.4),(97,1.9),(98,2.2),(99,6),(100,6.8)\n                        ) AS v(i, val)\n                    ), 10)\n                ))\",\n                None,\n            &[]).unwrap();\n\n            for _ in 0..10 {\n                let v = value_result.next().unwrap();\n                let t = tvec_result.next().unwrap();\n                assert_eq!(v[1].value::<&str>(), t[1].value::<&str>());\n                assert_eq!(v[2].value::<f64>().unwrap(), t[2].value::<f64>().unwrap());\n            }\n            assert!(value_result.next().is_none());\n            assert!(tvec_result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/bin/pgrx_embed.rs",
    "content": "// so we can support upgrading pgrx\n#![allow(unexpected_cfgs)]\n::pgrx::pgrx_embed!();\n"
  },
  {
    "path": "extension/src/candlestick.rs",
    "content": "use pgrx::*;\nuse serde::{Deserialize, Serialize};\n\nuse crate::accessors::{\n    AccessorClose, AccessorCloseTime, AccessorHigh, AccessorHighTime, AccessorLow, AccessorLowTime,\n    AccessorOpen, AccessorOpenTime,\n};\nuse crate::{\n    aggregate_utils::in_aggregate_context,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\nuse tspoint::TSPoint;\n\nflat_serialize_macro::flat_serialize! {\n    #[derive(Serialize, Deserialize, Debug, Copy)]\n    enum VolKind {\n        unused_but_required_by_flat_serialize: u64,\n        Missing: 1 {},\n        Transaction: 2 { vol: f64, vwap: f64 },\n    }\n}\n\npg_type! {\n    #[derive(Debug, Copy)]\n    struct Candlestick {\n        open: TSPoint,\n        high: TSPoint,\n        low: TSPoint,\n        close: TSPoint,\n        #[flat_serialize::flatten]\n        volume: VolKind,\n    }\n}\n\nimpl Candlestick {\n    pub fn new(ts: i64, open: f64, high: f64, low: f64, close: f64, volume: Option<f64>) -> Self {\n        let volume = match volume {\n            None => VolKind::Missing {},\n            Some(volume) => {\n                let typical = (high + low + close) / 3.0;\n                VolKind::Transaction {\n                    vol: volume,\n                    vwap: volume * typical,\n                }\n            }\n        };\n\n        unsafe {\n            flatten!(Candlestick {\n                open: TSPoint { ts, val: open },\n                high: TSPoint { ts, val: high },\n                low: TSPoint { ts, val: low },\n                close: TSPoint { ts, val: close },\n                volume,\n            })\n        }\n    }\n\n    pub fn from_tick(ts: i64, price: f64, volume: Option<f64>) -> Self {\n        Candlestick::new(ts, price, price, price, price, volume)\n    }\n\n    pub fn add_tick_data(&mut self, ts: i64, price: f64, volume: Option<f64>) {\n        if ts < self.open.ts {\n            self.open = TSPoint { ts, val: price };\n        }\n\n        if price > self.high.val {\n            self.high = TSPoint { ts, val: price };\n        }\n\n        if price < self.low.val {\n            self.low = TSPoint { ts, val: price };\n        }\n\n        if ts > self.close.ts {\n            self.close = TSPoint { ts, val: price };\n        }\n\n        if let (VolKind::Transaction { vol, vwap }, Some(volume)) = (self.volume, volume) {\n            self.volume = VolKind::Transaction {\n                vol: vol + volume,\n                vwap: vwap + volume * price,\n            };\n        } else {\n            self.volume = VolKind::Missing {};\n        };\n    }\n\n    pub fn combine(&mut self, candlestick: &Candlestick) {\n        if candlestick.open.ts < self.open.ts {\n            self.open = candlestick.open;\n        }\n\n        if candlestick.high.val > self.high.val {\n            self.high = candlestick.high;\n        }\n\n        if candlestick.low.val < self.low.val {\n            self.low = candlestick.low;\n        }\n\n        if candlestick.close.ts > self.close.ts {\n            self.close = candlestick.close;\n        }\n\n        if let (\n            VolKind::Transaction {\n                vol: vol1,\n                vwap: vwap1,\n            },\n            VolKind::Transaction {\n                vol: vol2,\n                vwap: vwap2,\n            },\n        ) = (self.volume, candlestick.volume)\n        {\n            self.volume = VolKind::Transaction {\n                vol: vol1 + vol2,\n                vwap: vwap1 + vwap2,\n            };\n        } else {\n            self.volume = VolKind::Missing {};\n        };\n    }\n\n    pub fn open(&self) -> f64 {\n        self.open.val\n    }\n\n    pub fn high(&self) -> f64 {\n        self.high.val\n    }\n\n    pub fn low(&self) -> f64 {\n        self.low.val\n    }\n\n    pub fn close(&self) -> f64 {\n        self.close.val\n    }\n\n    pub fn open_time(&self) -> i64 {\n        self.open.ts\n    }\n\n    pub fn high_time(&self) -> i64 {\n        self.high.ts\n    }\n\n    pub fn low_time(&self) -> i64 {\n        self.low.ts\n    }\n\n    pub fn close_time(&self) -> i64 {\n        self.close.ts\n    }\n\n    pub fn volume(&self) -> Option<f64> {\n        match self.volume {\n            VolKind::Transaction { vol, .. } => Some(vol),\n            VolKind::Missing {} => None,\n        }\n    }\n\n    pub fn vwap(&self) -> Option<f64> {\n        match self.volume {\n            VolKind::Transaction { vol, vwap } => {\n                if vol > 0.0 && vwap.is_finite() {\n                    Some(vwap / vol)\n                } else {\n                    None\n                }\n            }\n            VolKind::Missing {} => None,\n        }\n    }\n}\n\nron_inout_funcs!(Candlestick);\n\n#[pg_extern(immutable, parallel_safe)]\npub fn candlestick(\n    ts: Option<crate::raw::TimestampTz>,\n    open: Option<f64>,\n    high: Option<f64>,\n    low: Option<f64>,\n    close: Option<f64>,\n    volume: Option<f64>,\n) -> Option<Candlestick> {\n    match ts {\n        Some(ts) => Some(Candlestick::new(\n            ts.into(),\n            open?,\n            high?,\n            low?,\n            close?,\n            volume,\n        )),\n        None => None,\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn tick_data_no_vol_transition(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    price: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    tick_data_transition_inner(unsafe { state.to_inner() }, ts, price, None, fcinfo).internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn tick_data_transition(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    price: Option<f64>,\n    volume: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    tick_data_transition_inner(unsafe { state.to_inner() }, ts, price, volume, fcinfo).internal()\n}\n\npub fn tick_data_transition_inner(\n    state: Option<Inner<Candlestick>>,\n    ts: Option<crate::raw::TimestampTz>,\n    price: Option<f64>,\n    volume: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<Candlestick>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            if let (Some(ts), Some(price)) = (ts, price) {\n                match state {\n                    None => {\n                        let cs = Candlestick::from_tick(ts.into(), price, volume);\n                        Some(cs.into())\n                    }\n                    Some(mut cs) => {\n                        cs.add_tick_data(ts.into(), price, volume);\n                        Some(cs)\n                    }\n                }\n            } else {\n                state\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn candlestick_rollup_trans(\n    state: Internal,\n    value: Option<Candlestick>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    candlestick_rollup_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\n\npub fn candlestick_rollup_trans_inner(\n    state: Option<Inner<Candlestick>>,\n    value: Option<Candlestick>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<Candlestick>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, value) {\n            (state, None) => state,\n            (None, Some(value)) => Some(value.into()),\n            (Some(state), Some(value)) => {\n                let mut state = *state;\n                state.combine(&value);\n                Some(state.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn candlestick_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<Candlestick> {\n    unsafe { candlestick_final_inner(state.to_inner(), fcinfo) }\n}\n\npub fn candlestick_final_inner(\n    state: Option<Inner<Candlestick>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Candlestick> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let state = match state {\n                None => return None,\n                Some(state) => *state,\n            };\n            Some(state)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn candlestick_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { candlestick_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\n\npub fn candlestick_combine_inner(\n    state1: Option<Inner<Candlestick>>,\n    state2: Option<Inner<Candlestick>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<Candlestick>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(only)) | (Some(only), None) => Some((*only).into()),\n            (Some(a), Some(b)) => {\n                let (mut a, b) = (*a, *b);\n                a.combine(&b);\n                Some(a.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn candlestick_serialize(state: Internal) -> bytea {\n    let mut state = state;\n    let cs: &mut Candlestick = unsafe { state.get_mut().unwrap() };\n    let ser = &**cs;\n    crate::do_serialize!(ser)\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn candlestick_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    candlestick_deserialize_inner(bytes).internal()\n}\n\npub fn candlestick_deserialize_inner(bytes: bytea) -> Inner<Candlestick> {\n    let de: CandlestickData = crate::do_deserialize!(bytes, CandlestickData);\n    let cs: Candlestick = de.into();\n    cs.into()\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE candlestick_agg( \\n\\\n        ts TIMESTAMPTZ,\\n\\\n        price DOUBLE PRECISION,\\n\\\n        volume DOUBLE PRECISION\\n\\\n    )\\n\\\n    (\\n\\\n        sfunc = tick_data_transition,\\n\\\n        stype = internal,\\n\\\n        finalfunc = candlestick_final,\\n\\\n        combinefunc = candlestick_combine,\\n\\\n        serialfunc = candlestick_serialize,\\n\\\n        deserialfunc = candlestick_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\",\n    name = \"candlestick_agg\",\n    requires = [\n        tick_data_transition,\n        candlestick_final,\n        candlestick_combine,\n        candlestick_serialize,\n        candlestick_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup( candlestick Candlestick)\\n\\\n    (\\n\\\n        sfunc = candlestick_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = candlestick_final,\\n\\\n        combinefunc = candlestick_combine,\\n\\\n        serialfunc = candlestick_serialize,\\n\\\n        deserialfunc = candlestick_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\",\n    name = \"candlestick_rollup\",\n    requires = [\n        candlestick_rollup_trans,\n        candlestick_final,\n        candlestick_combine,\n        candlestick_serialize,\n        candlestick_deserialize\n    ],\n);\n\n#[pg_extern(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_open(candlestick: Option<Candlestick>, _accessor: AccessorOpen) -> Option<f64> {\n    candlestick.map(|cs| cs.open())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn open(candlestick: Option<Candlestick>) -> Option<f64> {\n    candlestick.map(|cs| cs.open())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_high(candlestick: Option<Candlestick>, _accessor: AccessorHigh) -> Option<f64> {\n    candlestick.map(|cs| cs.high())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn high(candlestick: Option<Candlestick>) -> Option<f64> {\n    candlestick.map(|cs| cs.high())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_low(candlestick: Option<Candlestick>, _accessor: AccessorLow) -> Option<f64> {\n    candlestick.map(|cs| cs.low())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn low(candlestick: Option<Candlestick>) -> Option<f64> {\n    candlestick.map(|cs| cs.low())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_close(candlestick: Option<Candlestick>, _accessor: AccessorClose) -> Option<f64> {\n    candlestick.map(|cs| cs.close())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn close(candlestick: Option<Candlestick>) -> Option<f64> {\n    candlestick.map(|cs| cs.close())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_open_time(\n    candlestick: Option<Candlestick>,\n    _accessor: AccessorOpenTime,\n) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.open_time().into())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn open_time(candlestick: Option<Candlestick>) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.open_time().into())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_high_time(\n    candlestick: Option<Candlestick>,\n    _accessor: AccessorHighTime,\n) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.high_time().into())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn high_time(candlestick: Option<Candlestick>) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.high_time().into())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_low_time(\n    candlestick: Option<Candlestick>,\n    _accessor: AccessorLowTime,\n) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.low_time().into())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn low_time(candlestick: Option<Candlestick>) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.low_time().into())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_close_time(\n    candlestick: Option<Candlestick>,\n    _accessor: AccessorCloseTime,\n) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.close_time().into())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn close_time(candlestick: Option<Candlestick>) -> Option<crate::raw::TimestampTz> {\n    candlestick.map(|cs| cs.close_time().into())\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn volume(candlestick: Option<Candlestick>) -> Option<f64> {\n    match candlestick {\n        None => None,\n        Some(cs) => cs.volume(),\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn vwap(candlestick: Option<Candlestick>) -> Option<f64> {\n    match candlestick {\n        None => None,\n        Some(cs) => cs.vwap(),\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use std::ptr;\n\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    macro_rules! select_one {\n        ($client:expr, $stmt:expr, $type:ty) => {\n            $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_one::<$type>()\n                .unwrap()\n        };\n    }\n    macro_rules! select_two {\n        ($client:expr, $stmt:expr, $type1:ty, $type2:ty) => {\n            $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_two::<$type1, $type2>()\n                .unwrap()\n        };\n    }\n\n    #[pg_test]\n    fn candlestick_single_point() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT candlestick(ts, open, high, low, close, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 0.0, 0.0, 0.0, 1.0)\n                          ) AS v(ts, open, high, low, close, volume)\"#;\n\n            let output = select_one!(client, stmt, &str);\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            close:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            volume:Transaction(vol:1,vwap:0)\\\n                            )\";\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_single_point() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT candlestick_agg(ts, price, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 1.0)\n                          ) AS v(ts, price, volume)\"#;\n\n            let output = select_one!(client, stmt, &str);\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            close:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            volume:Transaction(vol:1,vwap:0)\\\n                            )\";\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_accessors() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            for ohlc in [\"open\", \"high\", \"low\", \"close\"] {\n                let stmt = format!(\n                    r#\"SELECT\n                           {ohlc}(candlestick),\n                           {ohlc}_time(candlestick)::text\n                       FROM (\n                           SELECT candlestick(ts, open, high, low, close, volume)\n                           FROM (\n                               VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 0.0, 0.0, 0.0, 1.0)\n                           ) AS v(ts, open, high, low, close, volume)\n                       ) AS v(candlestick)\"#\n                );\n                let (val, ts) = select_two!(client, &stmt, f64, &str);\n                assert_eq!(0.0, val.unwrap());\n                assert_eq!(\"2022-08-01 00:00:00+00\", ts.unwrap());\n            }\n\n            // testing arrow operators\n            for ohlc in [\"open\", \"high\", \"low\", \"close\"] {\n                let stmt = format!(\n                    r#\"SELECT\n                           candlestick->{ohlc}(),\n                           (candlestick->{ohlc}_time())::text\n                       FROM (\n                           SELECT candlestick(ts, open, high, low, close, volume)\n                           FROM (\n                               VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 0.0, 0.0, 0.0, 1.0)\n                           ) AS v(ts, open, high, low, close, volume)\n                       ) AS v(candlestick)\"#\n                );\n                let (val, ts) = select_two!(client, &stmt, f64, &str);\n                assert_eq!(0.0, val.unwrap());\n                assert_eq!(\"2022-08-01 00:00:00+00\", ts.unwrap());\n            }\n\n            let stmt = r#\"SELECT\n                              volume(candlestick),\n                              vwap(candlestick)\n                          FROM (\n                              SELECT candlestick(ts, open, high, low, close, volume)\n                              FROM (\n                                  VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 0.0, 0.0, 0.0, 1.0)\n                              ) AS v(ts, open, high, low, close, volume)\n                          ) AS v(candlestick)\"#;\n            let (vol, vwap) = select_two!(client, stmt, f64, f64);\n            assert_eq!(1.0, vol.unwrap());\n            assert_eq!(0.0, vwap.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_accessors() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            for ohlc in [\"open\", \"high\", \"low\", \"close\"] {\n                let stmt = format!(\n                    r#\"SELECT\n                           {ohlc}(candlestick),\n                           {ohlc}_time(candlestick)::text\n                       FROM (\n                           SELECT candlestick_agg(ts, price, volume)\n                           FROM (\n                               VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 1.0)\n                           ) AS v(ts, price, volume)\n                       ) AS v(candlestick)\"#\n                );\n                let (val, ts) = select_two!(client, &stmt, f64, &str);\n                assert_eq!(0.0, val.unwrap());\n                assert_eq!(\"2022-08-01 00:00:00+00\", ts.unwrap());\n            }\n\n            let stmt = r#\"SELECT\n                               volume(candlestick),\n                               vwap(candlestick)\n                          FROM (\n                              SELECT candlestick_agg(ts, price, volume)\n                              FROM (\n                                  VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 1.0)\n                              ) AS v(ts, price, volume)\n                          ) AS v(candlestick)\"#;\n\n            let (vol, vwap) = select_two!(client, stmt, f64, f64);\n            assert_eq!(1.0, vol.unwrap());\n            assert_eq!(0.0, vwap.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_extreme_values() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            // timestamptz low and high val according to https://www.postgresql.org/docs/14/datatype-datetime.html\n            for extreme_time in &[\"4713-01-01 00:00:00+00 BC\", \"294276-12-31 23:59:59+00\"] {\n                let stmt = format!(\n                    r#\"SELECT candlestick_agg(ts, price, volume)::text\n                         FROM (VALUES ('{extreme_time}'::timestamptz, 1.0, 1.0)) AS v(ts, price, volume)\"#\n                );\n\n                let output = select_one!(client, &stmt, &str);\n\n                let expected = format!(\n                    \"(\\\n                            version:1,\\\n                            open:(ts:\\\"{extreme_time}\\\",val:1),\\\n                            high:(ts:\\\"{extreme_time}\\\",val:1),\\\n                            low:(ts:\\\"{extreme_time}\\\",val:1),\\\n                            close:(ts:\\\"{extreme_time}\\\",val:1),\\\n                            volume:Transaction(vol:1,vwap:1)\\\n                            )\"\n                );\n                assert_eq!(expected, output.unwrap());\n            }\n\n            for extreme_price in &[f64::MAX, f64::MIN] {\n                let stmt = format!(\n                    r#\"SELECT candlestick_agg(ts, price, volume)::text\n                 FROM (VALUES ('2022-08-01 00:00:00+00'::timestamptz, {extreme_price}, 1.0)) AS v(ts, price, volume)\"#\n                );\n\n                let output = select_one!(client, &stmt, &str);\n\n                let expected = format!(\n                    \"(\\\n                 version:1,\\\n                 open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:{extreme_price}),\\\n                 high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:{extreme_price}),\\\n                 low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:{extreme_price}),\\\n                 close:(ts:\\\"2022-08-01 00:00:00+00\\\",val:{extreme_price}),\\\n                 volume:Transaction(vol:1,vwap:{})\\\n                 )\",\n                    (extreme_price + extreme_price + extreme_price)\n                );\n                assert_eq!(expected, output.unwrap());\n            }\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_null_inputs() {\n        Spi::connect_mut(|client| {\n            for (t, o, h, l, c, v) in &[\n                (\"NULL\", \"NULL\", \"NULL\", \"NULL\", \"NULL\", \"NULL\"),\n                (\"NULL\", \"1.0\", \"1.0\", \"1.0\", \"1.0\", \"1.0\"),\n                (\"now()\", \"NULL\", \"1.0\", \"1.0\", \"1.0\", \"1.0\"),\n                (\"now()\", \"1.0\", \"NULL\", \"1.0\", \"1.0\", \"1.0\"),\n                (\"now()\", \"1.0\", \"1.0\", \"NULL\", \"1.0\", \"1.0\"),\n                (\"now()\", \"1.0\", \"1.0\", \"1.0\", \"NULL\", \"1.0\"),\n            ] {\n                let stmt = format!(\"SELECT candlestick({t}, {o}, {h}, {l}, {c}, {v})::TEXT\");\n                let output = select_one!(client, &stmt, String);\n                assert_eq!(output, None);\n            }\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_null_inputs() {\n        Spi::connect_mut(|client| {\n            for (ts, price, vol) in &[\n                (\"NULL\", \"NULL\", \"NULL\"),\n                (\"NULL\", \"1.0\", \"1.0\"),\n                (\"now()\", \"NULL\", \"1.0\"),\n            ] {\n                let stmt = format!(\"SELECT candlestick_agg({ts}, {price}, {vol})::text\");\n                let output = select_one!(client, &stmt, String);\n                assert_eq!(output, None);\n            }\n\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:1),\\\n                            high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:1),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:1),\\\n                            close:(ts:\\\"2022-08-01 00:00:00+00\\\",val:1),\\\n                            volume:Missing()\\\n                            )\";\n\n            let output = select_one!(\n                client,\n                \"SELECT candlestick_agg(ts, price, vol)::TEXT\n                   FROM (VALUES('2022-08-01 00:00:00+00'::timestamptz, 1.0, NULL::double precision)) AS v(ts, price, vol)\",\n                String\n            ).unwrap();\n            assert_eq!(expected, output);\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_as_constructor() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT\n                              candlestick(ts, open, high, low, close, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 0.0, 0.0, 0.0, 1.0),\n                                     ('2022-08-02 00:00:00+00'::timestamptz, 9.0, 12.0, 3.0, 6.0, 1.0)\n                          ) AS v(ts, open, high, low, close, volume)\"#;\n\n            let mut candlesticks = client.update(stmt, None, &[]).unwrap();\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            close:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            volume:Transaction(vol:1,vwap:0)\\\n                            )\";\n\n            assert_eq!(\n                Some(expected),\n                candlesticks.next().unwrap()[1].value().unwrap()\n            );\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-02 00:00:00+00\\\",val:9),\\\n                            high:(ts:\\\"2022-08-02 00:00:00+00\\\",val:12),\\\n                            low:(ts:\\\"2022-08-02 00:00:00+00\\\",val:3),\\\n                            close:(ts:\\\"2022-08-02 00:00:00+00\\\",val:6),\\\n                            volume:Transaction(vol:1,vwap:7)\\\n                            )\";\n\n            assert_eq!(\n                Some(expected),\n                candlesticks.next().unwrap()[1].value().unwrap()\n            );\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_constant() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT\n                              date_trunc('day', ts)::text,\n                              candlestick_agg(ts, price, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 1.0),\n                                     ('2022-08-01 06:00:00+00'::timestamptz, 0.0, 1.0),\n                                     ('2022-08-01 12:00:00+00'::timestamptz, 0.0, 1.0),\n                                     ('2022-08-01 18:00:00+00'::timestamptz, 0.0, 1.0),\n                                     ('2022-08-01 23:59:59+00'::timestamptz, 0.0, 1.0)\n                          ) AS v(ts, price, volume)\n                          GROUP BY 1\"#;\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            close:(ts:\\\"2022-08-01 23:59:59+00\\\",val:0),\\\n                            volume:Transaction(vol:5,vwap:0)\\\n                            )\";\n            let (_, output) = select_two!(client, stmt, &str, &str);\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_strictly_increasing() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT\n                              date_trunc('day', ts)::text,\n                              candlestick_agg(ts, price, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz, 1.0, 1.0),\n                                     ('2022-08-01 06:00:00+00'::timestamptz, 2.0, 1.0),\n                                     ('2022-08-01 12:00:00+00'::timestamptz, 3.0, 1.0),\n                                     ('2022-08-01 18:00:00+00'::timestamptz, 4.0, 1.0),\n                                     ('2022-08-01 23:59:59+00'::timestamptz, 5.0, 1.0)\n                          ) AS v(ts, price, volume)\n                          GROUP BY 1\"#;\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:1),\\\n                            high:(ts:\\\"2022-08-01 23:59:59+00\\\",val:5),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:1),\\\n                            close:(ts:\\\"2022-08-01 23:59:59+00\\\",val:5),\\\n                            volume:Transaction(vol:5,vwap:15)\\\n                            )\";\n            let (_, output) = select_two!(client, stmt, &str, &str);\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_strictly_decreasing() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT\n                              date_trunc('day', ts)::text,\n                              candlestick_agg(ts, price, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz, 5.0, 1.0),\n                                     ('2022-08-01 06:00:00+00'::timestamptz, 4.0, 1.0),\n                                     ('2022-08-01 12:00:00+00'::timestamptz, 3.0, 1.0),\n                                     ('2022-08-01 18:00:00+00'::timestamptz, 2.0, 1.0),\n                                     ('2022-08-01 23:59:59+00'::timestamptz, 1.0, 1.0)\n                          ) AS v(ts, price, volume)\n                          GROUP BY 1\"#;\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:5),\\\n                            high:(ts:\\\"2022-08-01 00:00:00+00\\\",val:5),\\\n                            low:(ts:\\\"2022-08-01 23:59:59+00\\\",val:1),\\\n                            close:(ts:\\\"2022-08-01 23:59:59+00\\\",val:1),\\\n                            volume:Transaction(vol:5,vwap:15)\\\n                            )\";\n            let (_, output) = select_two!(client, stmt, &str, &str);\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_oscillating() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"SELECT\n                              date_trunc('day', ts)::text,\n                              candlestick_agg(ts, price, volume)::text\n                          FROM (\n                              VALUES ('2022-08-01 00:00:00+00'::timestamptz,  3.0, 1.0),\n                                     ('2022-08-01 02:00:00+00'::timestamptz,  4.0, 1.0),\n                                     ('2022-08-01 04:00:00+00'::timestamptz, 11.0, 1.0),\n                                     ('2022-08-01 06:00:00+00'::timestamptz,  5.0, 1.0),\n                                     ('2022-08-01 08:00:00+00'::timestamptz,  2.0, 1.0),\n                                     ('2022-08-01 10:00:00+00'::timestamptz,  1.0, 1.0),\n                                     ('2022-08-01 12:00:00+00'::timestamptz, 12.0, 1.0),\n                                     ('2022-08-01 14:00:00+00'::timestamptz,  9.0, 1.0),\n                                     ('2022-08-01 16:00:00+00'::timestamptz, 10.0, 1.0),\n                                     ('2022-08-01 18:00:00+00'::timestamptz,  7.0, 1.0),\n                                     ('2022-08-01 20:00:00+00'::timestamptz,  6.0, 1.0),\n                                     ('2022-08-01 22:00:00+00'::timestamptz,  8.0, 1.0)\n                          ) AS v(ts, price, volume)\n                          GROUP BY 1\"#;\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:3),\\\n                            high:(ts:\\\"2022-08-01 12:00:00+00\\\",val:12),\\\n                            low:(ts:\\\"2022-08-01 10:00:00+00\\\",val:1),\\\n                            close:(ts:\\\"2022-08-01 22:00:00+00\\\",val:8),\\\n                            volume:Transaction(vol:12,vwap:78)\\\n                            )\";\n            let (_, output) = select_two!(client, stmt, &str, &str);\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_rollup() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"WITH t AS (\n                              SELECT\n                                  candlestick(ts, open, high, low, close, volume) AS candlestick\n                              FROM (\n                                  VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 4.0, 0.0, 4.0, 5.0),\n                                         ('2022-08-02 00:00:00+00'::timestamptz, 5.0, 8.0, 5.0, 8.0, 4.0)\n                              ) AS v(ts, open, high, low, close, volume)\n                          )\n                          SELECT\n                              rollup(candlestick)::text\n                          FROM t\"#;\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            high:(ts:\\\"2022-08-02 00:00:00+00\\\",val:8),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            close:(ts:\\\"2022-08-02 00:00:00+00\\\",val:8),\\\n                            volume:Transaction(vol:9,vwap:41.33333333333333)\\\n                            )\";\n\n            let output = select_one!(client, stmt, &str);\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_agg_rollup() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let stmt = r#\"WITH t AS (\n                              SELECT\n                                  date_trunc('day', ts) AS date,\n                                  candlestick_agg(ts, price, volume) AS candlestick\n                              FROM (\n                                  VALUES ('2022-08-01 00:00:00+00'::timestamptz, 0.0, 1.0),\n                                         ('2022-08-01 06:00:00+00'::timestamptz, 1.0, 1.0),\n                                         ('2022-08-01 12:00:00+00'::timestamptz, 2.0, 1.0),\n                                         ('2022-08-01 18:00:00+00'::timestamptz, 3.0, 1.0),\n                                         ('2022-08-01 23:59:59+00'::timestamptz, 4.0, 1.0),\n                                         ('2022-08-02 06:00:00+00'::timestamptz, 5.0, 1.0),\n                                         ('2022-08-02 12:00:00+00'::timestamptz, 6.0, 1.0),\n                                         ('2022-08-02 18:00:00+00'::timestamptz, 7.0, 1.0),\n                                         ('2022-08-02 23:59:59+00'::timestamptz, 8.0, 1.0)\n                              ) AS v(ts, price, volume)\n                              GROUP BY 1\n                          )\n                          SELECT\n                              date_trunc('month', date)::text,\n                              rollup(candlestick)::text\n                          FROM t\n                          GROUP BY 1\"#;\n\n            let expected = \"(\\\n                            version:1,\\\n                            open:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            high:(ts:\\\"2022-08-02 23:59:59+00\\\",val:8),\\\n                            low:(ts:\\\"2022-08-01 00:00:00+00\\\",val:0),\\\n                            close:(ts:\\\"2022-08-02 23:59:59+00\\\",val:8),\\\n                            volume:Transaction(vol:9,vwap:36)\\\n                            )\";\n            let (_, output) = select_two!(client, stmt, &str, &str);\n            assert_eq!(expected, output.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn candlestick_byte_io() {\n        let state = tick_data_transition_inner(\n            None,\n            Some(100.into()),\n            Some(10.0),\n            Some(1.0),\n            ptr::null_mut(),\n        );\n        let state = tick_data_transition_inner(\n            state,\n            Some(200.into()),\n            Some(1.0),\n            Some(2.0),\n            ptr::null_mut(),\n        );\n\n        let output_buffer = state.unwrap().to_pg_bytes();\n        let expected = [\n            128, 1, 0, 0, 1, 0, 0, 0, 100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36, 64, 100, 0,\n            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36, 64, 200, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            240, 63, 200, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 240, 63, 2, 0, 0, 0, 0, 0, 0, 0,\n            0, 0, 0, 0, 0, 0, 8, 64, 0, 0, 0, 0, 0, 0, 40, 64,\n        ];\n        assert_eq!(*output_buffer, expected);\n    }\n}\n"
  },
  {
    "path": "extension/src/counter_agg/accessors.rs",
    "content": "use pgrx::*;\n\nuse crate::{\n    counter_agg::{CounterSummary, CounterSummaryData, MetricSummary},\n    datum_utils::interval_to_ms,\n    pg_type, ron_inout_funcs,\n};\n\nuse tspoint::TSPoint;\n\npg_type! {\n    struct CounterInterpolatedRateAccessor {\n        timestamp : i64,\n        interval : i64,\n        prev : CounterSummaryData,\n        next : CounterSummaryData,\n        flags : u64,\n    }\n}\n\nron_inout_funcs!(CounterInterpolatedRateAccessor);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_rate\")]\nfn counter_interpolated_rate_accessor(\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: Option<CounterSummary>,\n    next: Option<CounterSummary>,\n) -> CounterInterpolatedRateAccessor {\n    fn empty_summary() -> Option<CounterSummary> {\n        let tmp = TSPoint { ts: 0, val: 0.0 };\n        let tmp = MetricSummary::new(&tmp, None);\n        let tmp = CounterSummary::from_internal_counter_summary(tmp);\n        Some(tmp)\n    }\n\n    let flags = u64::from(prev.is_some()) + if next.is_some() { 2 } else { 0 };\n    let prev = prev.or_else(empty_summary).unwrap().0;\n    let next = next.or_else(empty_summary).unwrap().0;\n    let interval = interval_to_ms(&start, &duration);\n    crate::build! {\n        CounterInterpolatedRateAccessor {\n            timestamp : start.into(),\n            interval,\n            prev,\n            next,\n            flags,\n        }\n    }\n}\n\npg_type! {\n    struct CounterInterpolatedDeltaAccessor {\n        timestamp : i64,\n        interval : i64,\n        prev : CounterSummaryData,\n        next : CounterSummaryData,\n        flags : u64,\n    }\n}\n\nron_inout_funcs!(CounterInterpolatedDeltaAccessor);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_delta\")]\nfn counter_interpolated_delta_accessor(\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: Option<CounterSummary>,\n    next: Option<CounterSummary>,\n) -> CounterInterpolatedDeltaAccessor {\n    fn empty_summary() -> Option<CounterSummary> {\n        let tmp = TSPoint { ts: 0, val: 0.0 };\n        let tmp = MetricSummary::new(&tmp, None);\n        let tmp = CounterSummary::from_internal_counter_summary(tmp);\n        Some(tmp)\n    }\n\n    let flags = u64::from(prev.is_some()) + if next.is_some() { 2 } else { 0 };\n    let prev = prev.or_else(empty_summary).unwrap().0;\n    let next = next.or_else(empty_summary).unwrap().0;\n    let interval = interval_to_ms(&start, &duration);\n    crate::build! {\n        CounterInterpolatedDeltaAccessor {\n            timestamp : start.into(),\n            interval,\n            prev,\n            next,\n            flags,\n        }\n    }\n}\n"
  },
  {
    "path": "extension/src/counter_agg.rs",
    "content": "use serde::{Deserialize, Serialize};\n\nuse pgrx::*;\n\nuse crate::{\n    accessors::{\n        AccessorCorr, AccessorCounterZeroTime, AccessorDelta, AccessorExtrapolatedDelta,\n        AccessorExtrapolatedRate, AccessorFirstTime, AccessorFirstVal, AccessorIdeltaLeft,\n        AccessorIdeltaRight, AccessorIntercept, AccessorIrateLeft, AccessorIrateRight,\n        AccessorLastTime, AccessorLastVal, AccessorNumChanges, AccessorNumElements,\n        AccessorNumResets, AccessorRate, AccessorSlope, AccessorTimeDelta, AccessorWithBounds,\n    },\n    aggregate_utils::in_aggregate_context,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    range::*,\n    ron_inout_funcs,\n};\n\nuse tspoint::TSPoint;\n\nuse counter_agg::{range::I64Range, CounterSummaryBuilder, MetricSummary};\nuse stats_agg::stats2d::StatsSummary2D;\n\nuse self::Method::*;\n\nuse crate::raw::tstzrange;\n\nuse crate::raw::bytea;\n\nmod accessors;\n\nuse accessors::{CounterInterpolatedDeltaAccessor, CounterInterpolatedRateAccessor};\n\n// pg_type! can't handle generics so use a type alias to specify the type for `stats`\ntype PgTypeHackStatsSummary2D = StatsSummary2D<f64>;\n// TODO wrap FlatSummary a la GaugeSummary - requires serialization version bump\npg_type! {\n    #[derive(Debug, PartialEq)]\n    struct CounterSummary {\n        stats: PgTypeHackStatsSummary2D,\n        first: TSPoint,\n        second: TSPoint,\n        penultimate:TSPoint,\n        last: TSPoint,\n        reset_sum: f64,\n        num_resets: u64,\n        num_changes: u64,\n        #[flat_serialize::flatten]\n        bounds: I64RangeWrapper,\n    }\n}\n\nron_inout_funcs!(CounterSummary);\n\nimpl CounterSummary {\n    pub fn to_internal_counter_summary(&self) -> MetricSummary {\n        MetricSummary {\n            first: self.first,\n            second: self.second,\n            penultimate: self.penultimate,\n            last: self.last,\n            reset_sum: self.reset_sum,\n            num_resets: self.num_resets,\n            num_changes: self.num_changes,\n            stats: self.stats,\n            bounds: self.bounds.to_i64range(),\n        }\n    }\n    pub fn from_internal_counter_summary(st: MetricSummary) -> Self {\n        unsafe {\n            flatten!(CounterSummary {\n                stats: st.stats,\n                first: st.first,\n                second: st.second,\n                penultimate: st.penultimate,\n                last: st.last,\n                reset_sum: st.reset_sum,\n                num_resets: st.num_resets,\n                num_changes: st.num_changes,\n                bounds: I64RangeWrapper::from_i64range(st.bounds)\n            })\n        }\n    }\n    // fn set_bounds(&mut self, bounds: Option<I64Range>){\n    //     self.bounds = &I64RangeWrapper::from_i64range(bounds);\n    // }\n    fn interpolate(\n        &self,\n        interval_start: i64,\n        interval_len: i64,\n        prev: Option<CounterSummary>,\n        next: Option<CounterSummary>,\n    ) -> CounterSummary {\n        let prev = if self.first.ts > interval_start {\n            prev.map(|summary| {\n                let first = if summary.last.val > self.first.val {\n                    TSPoint {\n                        ts: summary.last.ts,\n                        val: 0.,\n                    }\n                } else {\n                    summary.last\n                };\n                time_weighted_average::TimeWeightMethod::Linear\n                    .interpolate(first, Some(self.first), interval_start)\n                    .expect(\"unable to interpolate lower bound\")\n            })\n        } else {\n            None\n        };\n\n        let next = next.map(|summary| {\n            let last = if self.last.val > summary.first.val {\n                TSPoint {\n                    ts: self.last.ts,\n                    val: 0.,\n                }\n            } else {\n                self.last\n            };\n            time_weighted_average::TimeWeightMethod::Linear\n                .interpolate(last, Some(summary.first), interval_start + interval_len)\n                .expect(\"unable to interpolate upper bound\")\n        });\n\n        let builder = prev.map(|pt| CounterSummaryBuilder::new(&pt, None));\n        let mut builder = builder.map_or_else(\n            || {\n                let mut summary = self.clone();\n                summary.bounds = I64RangeWrapper::from_i64range(None);\n                summary.to_internal_counter_summary().into()\n            },\n            |mut builder| {\n                builder\n                    .combine(&self.to_internal_counter_summary())\n                    .expect(\"unable to add data to interpolation\");\n                builder\n            },\n        );\n\n        if let Some(next) = next {\n            builder\n                .add_point(&next)\n                .expect(\"unable to add final interpolated point\");\n        }\n\n        CounterSummary::from_internal_counter_summary(builder.build())\n    }\n}\n\n#[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]\npub struct CounterSummaryTransState {\n    #[serde(skip)]\n    point_buffer: Vec<TSPoint>,\n    #[serde(skip)]\n    bounds: Option<I64Range>, // stores bounds until we combine points, after which, the bounds are stored in each summary\n    // We have a summary buffer here in order to deal with the fact that when the cmobine function gets called it\n    // must first build up a buffer of InternalMetricSummaries, then sort them, then call the combine function in\n    // the correct order.\n    summary_buffer: Vec<MetricSummary>,\n}\n\nimpl CounterSummaryTransState {\n    fn new() -> Self {\n        Self {\n            point_buffer: vec![],\n            bounds: None,\n            summary_buffer: vec![],\n        }\n    }\n\n    fn push_point(&mut self, value: TSPoint) {\n        self.point_buffer.push(value);\n    }\n\n    // fn set_bounds(&mut self, bounds: Option<I64Range>){\n    //     self.bounds = bounds;\n    // }\n\n    fn combine_points(&mut self) {\n        if self.point_buffer.is_empty() {\n            return;\n        }\n        self.point_buffer.sort_unstable_by_key(|p| p.ts);\n        let mut iter = self.point_buffer.iter();\n        let mut summary = CounterSummaryBuilder::new(iter.next().unwrap(), self.bounds);\n        for p in iter {\n            summary\n                .add_point(p)\n                .unwrap_or_else(|e| pgrx::error!(\"{}\", e));\n        }\n        self.point_buffer.clear();\n        // TODO build method should check validity\n        // check bounds only after we've combined all the points, so we aren't doing it all the time.\n        if !summary.bounds_valid() {\n            panic!(\"counter bounds invalid\")\n        }\n        self.summary_buffer.push(summary.build());\n    }\n\n    fn push_summary(&mut self, other: &CounterSummaryTransState) {\n        let sum_iter = other.summary_buffer.iter();\n        for sum in sum_iter {\n            self.summary_buffer.push(sum.clone());\n        }\n    }\n\n    fn combine_summaries(&mut self) {\n        self.combine_points();\n\n        if self.summary_buffer.len() <= 1 {\n            return;\n        }\n        // TODO move much of this method to crate?\n        self.summary_buffer.sort_unstable_by_key(|s| s.first.ts);\n        let mut sum_iter = self.summary_buffer.iter();\n        let mut new_summary = CounterSummaryBuilder::from(sum_iter.next().unwrap().clone());\n        for sum in sum_iter {\n            new_summary\n                .combine(sum)\n                .unwrap_or_else(|e| pgrx::error!(\"{}\", e));\n        }\n        self.summary_buffer = vec![new_summary.build()];\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn counter_summary_trans_serialize(state: Internal) -> bytea {\n    let mut state = state;\n    let state: &mut CounterSummaryTransState = unsafe { state.get_mut().unwrap() };\n    state.combine_summaries();\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn counter_summary_trans_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    counter_summary_trans_deserialize_inner(bytes).internal()\n}\npub fn counter_summary_trans_deserialize_inner(bytes: bytea) -> Inner<CounterSummaryTransState> {\n    let c: CounterSummaryTransState = crate::do_deserialize!(bytes, CounterSummaryTransState);\n    c.into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn counter_agg_trans(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    bounds: Option<tstzrange>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    counter_agg_trans_inner(unsafe { state.to_inner() }, ts, val, bounds, fcinfo).internal()\n}\npub fn counter_agg_trans_inner(\n    state: Option<Inner<CounterSummaryTransState>>,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    bounds: Option<tstzrange>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<CounterSummaryTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let p = match (ts, val) {\n                (_, None) => return state,\n                (None, _) => return state,\n                (Some(ts), Some(val)) => TSPoint { ts: ts.into(), val },\n            };\n            match state {\n                None => {\n                    let mut s = CounterSummaryTransState::new();\n                    if let Some(r) = bounds {\n                        s.bounds = get_range(r.0.cast_mut_ptr());\n                    }\n                    s.push_point(p);\n                    Some(s.into())\n                }\n                Some(mut s) => {\n                    s.push_point(p);\n                    Some(s)\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn counter_agg_trans_no_bounds(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    counter_agg_trans_inner(unsafe { state.to_inner() }, ts, val, None, fcinfo).internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn counter_agg_summary_trans(\n    state: Internal,\n    value: Option<CounterSummary>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    counter_agg_summary_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn counter_agg_summary_trans_inner(\n    state: Option<Inner<CounterSummaryTransState>>,\n    value: Option<CounterSummary>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<CounterSummaryTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, value) {\n            (state, None) => state,\n            (None, Some(value)) => {\n                let mut state = CounterSummaryTransState::new();\n                state\n                    .summary_buffer\n                    .push(value.to_internal_counter_summary());\n                Some(state.into())\n            }\n            (Some(mut state), Some(value)) => {\n                state\n                    .summary_buffer\n                    .push(value.to_internal_counter_summary());\n                Some(state)\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn counter_agg_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { counter_agg_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\npub fn counter_agg_combine_inner(\n    state1: Option<Inner<CounterSummaryTransState>>,\n    state2: Option<Inner<CounterSummaryTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<CounterSummaryTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state1, state2) {\n                (None, None) => None,\n                (None, Some(state2)) => {\n                    let mut s = state2.clone();\n                    s.combine_points();\n                    Some(s.into())\n                }\n                (Some(state1), None) => {\n                    let mut s = state1.clone();\n                    s.combine_points();\n                    Some(s.into())\n                } //should I make these return themselves?\n                (Some(state1), Some(state2)) => {\n                    let mut s1 = state1.clone(); // is there a way to avoid if it doesn't need it?\n                    s1.combine_points();\n                    let mut s2 = state2.clone();\n                    s2.combine_points();\n                    s2.push_summary(&s1);\n                    Some(s2.into())\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn counter_agg_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<CounterSummary> {\n    counter_agg_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\nfn counter_agg_final_inner(\n    state: Option<Inner<CounterSummaryTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<CounterSummary> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state.clone(),\n            };\n            state.combine_summaries();\n            debug_assert!(state.summary_buffer.len() <= 1);\n            match state.summary_buffer.pop() {\n                None => None,\n                Some(st) => {\n                    // there are some edge cases that this should prevent, but I'm not sure it's necessary, we do check the bounds in the functions that use them.\n                    if !st.bounds_valid() {\n                        panic!(\"counter bounds invalid\")\n                    }\n                    Some(CounterSummary::from_internal_counter_summary(st))\n                }\n            }\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE counter_agg( ts timestamptz, value DOUBLE PRECISION, bounds tstzrange )\\n\\\n    (\\n\\\n        sfunc = counter_agg_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = counter_agg_final,\\n\\\n        combinefunc = counter_agg_combine,\\n\\\n        serialfunc = counter_summary_trans_serialize,\\n\\\n        deserialfunc = counter_summary_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\",\n    name = \"counter_agg\",\n    requires = [\n        counter_agg_trans,\n        counter_agg_final,\n        counter_agg_combine,\n        counter_summary_trans_serialize,\n        counter_summary_trans_deserialize\n    ],\n);\n\n// allow calling counter agg without bounds provided.\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE counter_agg( ts timestamptz, value DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = counter_agg_trans_no_bounds,\\n\\\n        stype = internal,\\n\\\n        finalfunc = counter_agg_final,\\n\\\n        combinefunc = counter_agg_combine,\\n\\\n        serialfunc = counter_summary_trans_serialize,\\n\\\n        deserialfunc = counter_summary_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\\\n\",\n    name = \"counter_agg2\",\n    requires = [\n        counter_agg_trans_no_bounds,\n        counter_agg_final,\n        counter_agg_combine,\n        counter_summary_trans_serialize,\n        counter_summary_trans_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(cs CounterSummary)\\n\\\n    (\\n\\\n        sfunc = counter_agg_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = counter_agg_final,\\n\\\n        combinefunc = counter_agg_combine,\\n\\\n        serialfunc = counter_summary_trans_serialize,\\n\\\n        deserialfunc = counter_summary_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\\\n\",\n    name = \"counter_rollup\",\n    requires = [\n        counter_agg_summary_trans,\n        counter_agg_final,\n        counter_agg_combine,\n        counter_summary_trans_serialize,\n        counter_summary_trans_deserialize\n    ],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_delta(sketch: CounterSummary, _accessor: AccessorDelta) -> f64 {\n    counter_agg_delta(sketch)\n}\n\n#[pg_extern(name = \"delta\", strict, immutable, parallel_safe)]\nfn counter_agg_delta(summary: CounterSummary) -> f64 {\n    summary.to_internal_counter_summary().delta()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_rate(sketch: CounterSummary, _accessor: AccessorRate) -> Option<f64> {\n    counter_agg_rate(sketch)\n}\n\n#[pg_extern(name = \"rate\", strict, immutable, parallel_safe)]\nfn counter_agg_rate(summary: CounterSummary) -> Option<f64> {\n    summary.to_internal_counter_summary().rate()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_time_delta(sketch: CounterSummary, _accessor: AccessorTimeDelta) -> f64 {\n    counter_agg_time_delta(sketch)\n}\n\n#[pg_extern(name = \"time_delta\", strict, immutable, parallel_safe)]\nfn counter_agg_time_delta(summary: CounterSummary) -> f64 {\n    summary.to_internal_counter_summary().time_delta()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_irate_left(\n    sketch: CounterSummary,\n    _accessor: AccessorIrateLeft,\n) -> Option<f64> {\n    counter_agg_irate_left(sketch)\n}\n\n#[pg_extern(name = \"irate_left\", strict, immutable, parallel_safe)]\nfn counter_agg_irate_left(summary: CounterSummary) -> Option<f64> {\n    summary.to_internal_counter_summary().irate_left()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_irate_right(\n    sketch: CounterSummary,\n    _accessor: AccessorIrateRight,\n) -> Option<f64> {\n    counter_agg_irate_right(sketch)\n}\n\n#[pg_extern(name = \"irate_right\", strict, immutable, parallel_safe)]\nfn counter_agg_irate_right(summary: CounterSummary) -> Option<f64> {\n    summary.to_internal_counter_summary().irate_right()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_idelta_left(sketch: CounterSummary, _accessor: AccessorIdeltaLeft) -> f64 {\n    counter_agg_idelta_left(sketch)\n}\n\n#[pg_extern(name = \"idelta_left\", strict, immutable, parallel_safe)]\nfn counter_agg_idelta_left(summary: CounterSummary) -> f64 {\n    summary.to_internal_counter_summary().idelta_left()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_idelta_right(\n    sketch: CounterSummary,\n    _accessor: AccessorIdeltaRight,\n) -> f64 {\n    counter_agg_idelta_right(sketch)\n}\n\n#[pg_extern(name = \"idelta_right\", strict, immutable, parallel_safe)]\nfn counter_agg_idelta_right(summary: CounterSummary) -> f64 {\n    summary.to_internal_counter_summary().idelta_right()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_with_bounds(\n    sketch: CounterSummary,\n    accessor: AccessorWithBounds,\n) -> CounterSummary {\n    let mut builder = CounterSummaryBuilder::from(sketch.to_internal_counter_summary());\n    builder.set_bounds(accessor.bounds());\n    CounterSummary::from_internal_counter_summary(builder.build())\n}\n\n#[pg_extern(name = \"with_bounds\", strict, immutable, parallel_safe)]\nfn counter_agg_with_bounds(summary: CounterSummary, bounds: tstzrange) -> CounterSummary {\n    // TODO dedup with previous by using apply_bounds\n    unsafe {\n        let ptr = bounds.0.cast_mut_ptr();\n        let mut builder = CounterSummaryBuilder::from(summary.to_internal_counter_summary());\n        builder.set_bounds(get_range(ptr));\n        CounterSummary::from_internal_counter_summary(builder.build())\n    }\n}\n\n// TODO MetricSummary::with_bounds ?\n//     fn with_bounds(mut self, bounds: Option<I64Range>) -> Self {\n//         self.bounds = bounds;\n//         self\n//     }\n// fn apply_bounds(summary: MetricSummary, bounds: Option<I64Range>) -> MetricSummary {\n//     let mut builder = CounterSummaryBuilder::from(summary.to_internal_counter_summary());\n//     builder.set_bounds(bounds);\n//     CounterSummary::from_internal_counter_summary(builder.build())\n// }\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_extrapolated_delta(\n    sketch: CounterSummary,\n    accessor: AccessorExtrapolatedDelta,\n) -> Option<f64> {\n    counter_agg_extrapolated_delta(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"extrapolated_delta\", strict, immutable, parallel_safe)]\nfn counter_agg_extrapolated_delta(summary: CounterSummary, method: &str) -> Option<f64> {\n    match method_kind(method) {\n        Prometheus => summary\n            .to_internal_counter_summary()\n            .prometheus_delta()\n            .unwrap(),\n    }\n}\n\n#[pg_extern(name = \"interpolated_delta\", immutable, parallel_safe)]\nfn counter_agg_interpolated_delta(\n    summary: CounterSummary,\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: Option<CounterSummary>,\n    next: Option<CounterSummary>,\n) -> f64 {\n    let interval = crate::datum_utils::interval_to_ms(&start, &duration);\n    summary\n        .interpolate(start.into(), interval, prev, next)\n        .to_internal_counter_summary()\n        .delta()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_interpolated_delta(\n    sketch: CounterSummary,\n    accessor: CounterInterpolatedDeltaAccessor,\n) -> f64 {\n    let prev = if accessor.flags & 1 == 1 {\n        Some(accessor.prev.clone().into())\n    } else {\n        None\n    };\n    let next = if accessor.flags & 2 == 2 {\n        Some(accessor.next.clone().into())\n    } else {\n        None\n    };\n\n    counter_agg_interpolated_delta(\n        sketch,\n        accessor.timestamp.into(),\n        accessor.interval.into(),\n        prev,\n        next,\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_extrapolated_rate(\n    sketch: CounterSummary,\n    accessor: AccessorExtrapolatedRate,\n) -> Option<f64> {\n    counter_agg_extrapolated_rate(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"extrapolated_rate\", strict, immutable, parallel_safe)]\nfn counter_agg_extrapolated_rate(summary: CounterSummary, method: &str) -> Option<f64> {\n    match method_kind(method) {\n        Prometheus => summary\n            .to_internal_counter_summary()\n            .prometheus_rate()\n            .unwrap(),\n    }\n}\n\n#[pg_extern(name = \"interpolated_rate\", immutable, parallel_safe)]\nfn counter_agg_interpolated_rate(\n    summary: CounterSummary,\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: Option<CounterSummary>,\n    next: Option<CounterSummary>,\n) -> Option<f64> {\n    let interval = crate::datum_utils::interval_to_ms(&start, &duration);\n    summary\n        .interpolate(start.into(), interval, prev, next)\n        .to_internal_counter_summary()\n        .rate()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_interpolated_rate(\n    sketch: CounterSummary,\n    accessor: CounterInterpolatedRateAccessor,\n) -> Option<f64> {\n    let prev = if accessor.flags & 1 == 1 {\n        Some(accessor.prev.clone().into())\n    } else {\n        None\n    };\n    let next = if accessor.flags & 2 == 2 {\n        Some(accessor.next.clone().into())\n    } else {\n        None\n    };\n\n    counter_agg_interpolated_rate(\n        sketch,\n        accessor.timestamp.into(),\n        accessor.interval.into(),\n        prev,\n        next,\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_num_elements(\n    sketch: CounterSummary,\n    _accessor: AccessorNumElements,\n) -> i64 {\n    counter_agg_num_elements(sketch)\n}\n\n#[pg_extern(name = \"num_elements\", strict, immutable, parallel_safe)]\nfn counter_agg_num_elements(summary: CounterSummary) -> i64 {\n    summary.to_internal_counter_summary().stats.n as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_num_changes(sketch: CounterSummary, _accessor: AccessorNumChanges) -> i64 {\n    counter_agg_num_changes(sketch)\n}\n\n#[pg_extern(name = \"num_changes\", strict, immutable, parallel_safe)]\nfn counter_agg_num_changes(summary: CounterSummary) -> i64 {\n    summary.to_internal_counter_summary().num_changes as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_num_resets(sketch: CounterSummary, _accessor: AccessorNumResets) -> i64 {\n    counter_agg_num_resets(sketch)\n}\n\n#[pg_extern(name = \"num_resets\", strict, immutable, parallel_safe)]\nfn counter_agg_num_resets(summary: CounterSummary) -> i64 {\n    summary.to_internal_counter_summary().num_resets as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_slope(sketch: CounterSummary, _accessor: AccessorSlope) -> Option<f64> {\n    counter_agg_slope(sketch)\n}\n\n#[pg_extern(name = \"slope\", strict, immutable, parallel_safe)]\nfn counter_agg_slope(summary: CounterSummary) -> Option<f64> {\n    summary.to_internal_counter_summary().stats.slope()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_intercept(\n    sketch: CounterSummary,\n    _accessor: AccessorIntercept,\n) -> Option<f64> {\n    counter_agg_intercept(sketch)\n}\n\n#[pg_extern(name = \"intercept\", strict, immutable, parallel_safe)]\nfn counter_agg_intercept(summary: CounterSummary) -> Option<f64> {\n    summary.to_internal_counter_summary().stats.intercept()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_corr(sketch: CounterSummary, _accessor: AccessorCorr) -> Option<f64> {\n    counter_agg_corr(sketch)\n}\n\n#[pg_extern(name = \"corr\", strict, immutable, parallel_safe)]\nfn counter_agg_corr(summary: CounterSummary) -> Option<f64> {\n    summary.to_internal_counter_summary().stats.corr()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_zero_time(\n    sketch: CounterSummary,\n    _accessor: AccessorCounterZeroTime,\n) -> Option<crate::raw::TimestampTz> {\n    counter_agg_counter_zero_time(sketch)\n}\n\n#[pg_extern(name = \"counter_zero_time\", strict, immutable, parallel_safe)]\nfn counter_agg_counter_zero_time(summary: CounterSummary) -> Option<crate::raw::TimestampTz> {\n    Some(((summary.to_internal_counter_summary().stats.x_intercept()? * 1_000_000.0) as i64).into())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_first_val(sketch: CounterSummary, _accessor: AccessorFirstVal) -> f64 {\n    counter_agg_first_val(sketch)\n}\n\n#[pg_extern(name = \"first_val\", strict, immutable, parallel_safe)]\nfn counter_agg_first_val(summary: CounterSummary) -> f64 {\n    summary.to_internal_counter_summary().first.val\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_last_val(sketch: CounterSummary, _accessor: AccessorLastVal) -> f64 {\n    counter_agg_last_val(sketch)\n}\n\n#[pg_extern(name = \"last_val\", strict, immutable, parallel_safe)]\nfn counter_agg_last_val(summary: CounterSummary) -> f64 {\n    summary.to_internal_counter_summary().last.val\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_first_time(\n    sketch: CounterSummary,\n    _accessor: AccessorFirstTime,\n) -> crate::raw::TimestampTz {\n    counter_agg_first_time(sketch)\n}\n\n#[pg_extern(name = \"first_time\", strict, immutable, parallel_safe)]\nfn counter_agg_first_time(summary: CounterSummary) -> crate::raw::TimestampTz {\n    summary.to_internal_counter_summary().first.ts.into()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_counter_agg_last_time(\n    sketch: CounterSummary,\n    _accessor: AccessorLastTime,\n) -> crate::raw::TimestampTz {\n    counter_agg_last_time(sketch)\n}\n\n#[pg_extern(name = \"last_time\", strict, immutable, parallel_safe)]\nfn counter_agg_last_time(summary: CounterSummary) -> crate::raw::TimestampTz {\n    summary.to_internal_counter_summary().last.ts.into()\n}\n\n#[derive(\n    Clone, Copy, Debug, serde::Serialize, serde::Deserialize, flat_serialize_macro::FlatSerializable,\n)]\n#[repr(u8)]\npub enum Method {\n    Prometheus = 1,\n}\n\nimpl Method {\n    pub fn as_str(&self) -> &'static str {\n        match self {\n            Method::Prometheus => \"prometheus\",\n        }\n    }\n}\n\n#[track_caller]\npub fn method_kind(method: &str) -> Method {\n    match as_method(method) {\n        Some(method) => method,\n        None => pgrx::error!(\"unknown analysis method. Valid methods are 'prometheus'\"),\n    }\n}\n\npub fn as_method(method: &str) -> Option<Method> {\n    match method.trim().to_lowercase().as_str() {\n        \"prometheus\" => Some(Method::Prometheus),\n        _ => None,\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n\n    use super::testing::*;\n    use super::*;\n    use approx::assert_relative_eq;\n\n    macro_rules! select_one {\n        ($client:expr, $stmt:expr, $type:ty) => {\n            $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_one::<$type>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    macro_rules! select_and_check_one {\n        ($client:expr, $stmt:expr, $type:ty) => {{\n            let (a, b) = $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_two::<$type, $type>()\n                .unwrap();\n            assert_eq!(a, b);\n            a.unwrap()\n        }};\n    }\n\n    //do proper numerical comparisons on the values where that matters, use exact where it should be exact.\n    // copied from counter_agg crate\n    #[track_caller]\n    fn assert_close_enough(p1: &MetricSummary, p2: &MetricSummary) {\n        assert_eq!(p1.first, p2.first, \"first\");\n        assert_eq!(p1.second, p2.second, \"second\");\n        assert_eq!(p1.penultimate, p2.penultimate, \"penultimate\");\n        assert_eq!(p1.last, p2.last, \"last\");\n        assert_eq!(p1.num_changes, p2.num_changes, \"num_changes\");\n        assert_eq!(p1.num_resets, p2.num_resets, \"num_resets\");\n        assert_eq!(p1.stats.n, p2.stats.n, \"n\");\n        assert_relative_eq!(p1.stats.sx, p2.stats.sx);\n        assert_relative_eq!(p1.stats.sx2, p2.stats.sx2);\n        assert_relative_eq!(p1.stats.sy, p2.stats.sy);\n        assert_relative_eq!(p1.stats.sy2, p2.stats.sy2);\n        assert_relative_eq!(p1.stats.sxy, p2.stats.sxy);\n    }\n\n    #[pg_test]\n    fn test_counter_aggregate() {\n        Spi::connect_mut(|client| {\n            // set search_path after defining our table so we don't pollute the wrong schema\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            let stmt = \"SELECT format('toolkit_experimental, %s',current_setting('search_path'))\";\n            let search_path = select_one!(client, stmt, String);\n            client\n                .update(\n                    &format!(\"SET LOCAL search_path TO {search_path}\"),\n                    None,\n                    &[],\n                )\n                .unwrap();\n            make_test_table(client, \"test\");\n\n            // NULL bounds are equivalent to none provided\n            let stmt = \"SELECT counter_agg(ts, val) FROM test\";\n            let a = select_one!(client, stmt, CounterSummary);\n            let stmt = \"SELECT counter_agg(ts, val, NULL::tstzrange) FROM test\";\n            let b = select_one!(client, stmt, CounterSummary);\n            assert_close_enough(\n                &a.to_internal_counter_summary(),\n                &b.to_internal_counter_summary(),\n            );\n\n            let stmt = \"SELECT \\\n                delta(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->delta() \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 10.0);\n\n            let stmt = \"SELECT \\\n                time_delta(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->time_delta() \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 60.0);\n\n            // have to add 1 ms to right bounds to get full range and simple values because prometheus subtracts a ms\n            let stmt = \"SELECT \\\n                extrapolated_delta(counter_agg(ts, val, '[2020-01-01 00:00:00+00, 2020-01-01 00:02:00.001+00)'), 'prometheus'), \\\n                counter_agg(ts, val, '[2020-01-01 00:00:00+00, 2020-01-01 00:02:00.001+00)') -> extrapolated_delta('prometheus')  \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 20.0);\n            // doesn't matter if we set the bounds before or after\n            let stmt = \"SELECT \\\n                extrapolated_delta(with_bounds(counter_agg(ts, val), '[2020-01-01 00:00:00+00, 2020-01-01 00:02:00.001+00)'), 'prometheus'), \\\n                counter_agg(ts, val)->with_bounds('[2020-01-01 00:00:00+00, 2020-01-01 00:02:00.001+00)')-> extrapolated_delta('prometheus') \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 20.0);\n\n            let stmt = \"SELECT \\\n                extrapolated_rate(counter_agg(ts, val, '[2020-01-01 00:00:00+00, 2020-01-01 00:02:00.001+00)'), 'prometheus'), \\\n                counter_agg(ts, val, '[2020-01-01 00:00:00+00, 2020-01-01 00:02:00.001+00)')->extrapolated_rate('prometheus') \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 20.0 / 120.0);\n\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:02:00+00', 10.0), ('2020-01-01 00:03:00+00', 20.0), ('2020-01-01 00:04:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"SELECT \\\n                slope(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->slope() \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 10.0 / 60.0);\n\n            let stmt = \"SELECT \\\n                intercept(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->intercept() \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), -105191990.0);\n\n            let stmt = \"SELECT \\\n                corr(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->corr() \\\n            FROM test\";\n            assert_relative_eq!(select_and_check_one!(client, stmt, f64), 1.0);\n\n            let stmt = \"SELECT \\\n                counter_zero_time(counter_agg(ts, val))::TEXT, \\\n                (counter_agg(ts, val)->counter_zero_time())::TEXT \\\n            FROM test\";\n            let zp = select_and_check_one!(client, stmt, String);\n            assert_eq!(&zp, \"2019-12-31 23:59:00+00\");\n\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:08:00+00', 30.0), ('2020-01-01 00:10:00+00', 30.0), ('2020-01-01 00:10:30+00', 10.0), ('2020-01-01 00:20:00+00', 40.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"SELECT \\\n                num_elements(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->num_elements() \\\n            FROM test\";\n            assert_eq!(select_and_check_one!(client, stmt, i64), 9);\n\n            let stmt = \"SELECT \\\n                num_resets(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->num_resets() \\\n            FROM test\";\n            assert_eq!(select_and_check_one!(client, stmt, i64), 3);\n\n            let stmt = \"SELECT \\\n                num_changes(counter_agg(ts, val)), \\\n                counter_agg(ts, val)->num_changes() \\\n            FROM test\";\n            assert_eq!(select_and_check_one!(client, stmt, i64), 7);\n\n            //combine function works as expected\n            let stmt = \"SELECT counter_agg(ts, val) FROM test\";\n            let a = select_one!(client, stmt, CounterSummary);\n            let stmt = \"WITH t as (SELECT date_trunc('minute', ts), counter_agg(ts, val) as agg FROM test group by 1 ) SELECT rollup(agg) FROM t\";\n            let b = select_one!(client, stmt, CounterSummary);\n            assert_close_enough(\n                &a.to_internal_counter_summary(),\n                &b.to_internal_counter_summary(),\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_counter_io() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n            let stmt = \"INSERT INTO test VALUES\\\n                ('2020-01-01 00:00:00+00', 10.0),\\\n                ('2020-01-01 00:01:00+00', 20.0),\\\n                ('2020-01-01 00:02:00+00', 30.0),\\\n                ('2020-01-01 00:03:00+00', 20.0),\\\n                ('2020-01-01 00:04:00+00', 10.0),\\\n                ('2020-01-01 00:05:00+00', 20.0),\\\n                ('2020-01-01 00:06:00+00', 10.0),\\\n                ('2020-01-01 00:07:00+00', 30.0),\\\n                ('2020-01-01 00:08:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                stats:(\\\n                    n:9,\\\n                    sx:5680370160,\\\n                    sx2:216000,\\\n                    sx3:0,\\\n                    sx4:9175680000,\\\n                    sy:530,\\\n                    sy2:9688.888888888889,\\\n                    sy3:13308.641975308623,\\\n                    sy4:18597366.255144034,\\\n                    sxy:45600\\\n                ),\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                second:(ts:\\\"2020-01-01 00:01:00+00\\\",val:20),\\\n                penultimate:(ts:\\\"2020-01-01 00:07:00+00\\\",val:30),\\\n                last:(ts:\\\"2020-01-01 00:08:00+00\\\",val:10),\\\n                reset_sum:100,\\\n                num_resets:4,\\\n                num_changes:8,\\\n                bounds:(\\\n                    is_present:0,\\\n                    has_left:0,\\\n                    has_right:0,\\\n                    padding:(0,0,0,0,0),\\\n                    left:None,\\\n                    right:None\\\n                )\\\n            )\";\n\n            let stmt = \"SELECT counter_agg(ts, val)::TEXT FROM test\";\n            let test = select_one!(client, stmt, String);\n            assert_eq!(test, expected);\n\n            let stmt = format!(\"SELECT '{expected}'::CounterSummary::TEXT\");\n            let round_trip = select_one!(client, &stmt, String);\n            assert_eq!(expected, round_trip);\n\n            let stmt = \"SELECT delta(counter_agg(ts, val)) FROM test\";\n            let delta = select_one!(client, stmt, f64);\n            assert!((delta - 100.).abs() < f64::EPSILON);\n            let stmt = format!(\"SELECT delta('{expected}')\");\n            let delta_test = select_one!(client, &stmt, f64);\n            assert!((delta - delta_test).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT num_resets(counter_agg(ts, val)) FROM test\";\n            let resets = select_one!(client, stmt, i64);\n            assert_eq!(resets, 4);\n            let stmt = format!(\"SELECT num_resets('{expected}')\");\n            let resets_test = select_one!(client, &stmt, i64);\n            assert_eq!(resets, resets_test);\n        });\n    }\n\n    #[pg_test]\n    fn test_counter_byte_io() {\n        unsafe {\n            use std::ptr;\n            const BASE: i64 = 631152000000000;\n            const MIN: i64 = 60000000;\n            let state =\n                counter_agg_trans_inner(None, Some(BASE.into()), Some(10.0), None, ptr::null_mut());\n            let state = counter_agg_trans_inner(\n                state,\n                Some((BASE + MIN).into()),\n                Some(20.0),\n                None,\n                ptr::null_mut(),\n            );\n            let state = counter_agg_trans_inner(\n                state,\n                Some((BASE + 2 * MIN).into()),\n                Some(30.0),\n                None,\n                ptr::null_mut(),\n            );\n            let state = counter_agg_trans_inner(\n                state,\n                Some((BASE + 3 * MIN).into()),\n                Some(10.0),\n                None,\n                ptr::null_mut(),\n            );\n            let state = counter_agg_trans_inner(\n                state,\n                Some((BASE + 4 * MIN).into()),\n                Some(20.0),\n                None,\n                ptr::null_mut(),\n            );\n            let state = counter_agg_trans_inner(\n                state,\n                Some((BASE + 5 * MIN).into()),\n                Some(30.0),\n                None,\n                ptr::null_mut(),\n            );\n\n            let mut control = state.unwrap();\n            let buffer =\n                counter_summary_trans_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = pgrx::varlena::varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let expected = [\n                1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 96, 194, 134, 7, 62, 2, 0, 0, 0, 0, 0, 0, 0, 36,\n                64, 0, 231, 85, 138, 7, 62, 2, 0, 0, 0, 0, 0, 0, 0, 52, 64, 0, 124, 16, 149, 7, 62,\n                2, 0, 0, 0, 0, 0, 0, 0, 52, 64, 0, 3, 164, 152, 7, 62, 2, 0, 0, 0, 0, 0, 0, 0, 62,\n                64, 0, 0, 0, 0, 0, 0, 62, 64, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0,\n                0, 0, 0, 0, 0, 0, 0, 0, 128, 144, 246, 54, 236, 65, 0, 0, 0, 0, 0, 195, 238, 64, 0,\n                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 24, 32, 17, 209, 65, 0, 0, 0, 0, 0, 64, 106, 64, 0,\n                0, 0, 0, 0, 88, 155, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 76, 248, 42, 65, 0, 0,\n                0, 0, 0, 130, 196, 64, 0,\n            ];\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let new_state = counter_summary_trans_deserialize_inner(bytea(pg_sys::Datum::from(\n                expected.as_ptr(),\n            )));\n\n            control.combine_summaries(); // Serialized form is always combined\n            assert_eq!(&*new_state, &*control);\n        }\n    }\n\n    #[pg_test]\n    fn delta_after_counter_decrease() {\n        Spi::connect_mut(|client| {\n            decrease(client);\n            let stmt = \"SELECT delta(counter_agg(ts, val)) FROM test\";\n            // 10 after 30 means there was a reset so we add 30 + 10 = 40.\n            // Delta from 30 to 40 => 10\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_counter_increase() {\n        Spi::connect_mut(|client| {\n            increase(client);\n            let stmt = \"SELECT delta(counter_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_counter_decrease_then_increase_to_same_value() {\n        Spi::connect_mut(|client| {\n            decrease_then_increase_to_same_value(client);\n            let stmt = \"SELECT delta(counter_agg(ts, val)) FROM test\";\n            // 10 after 30 means there was a reset so we add 30 + 10 + 30 = 70.\n            // Delta from 30 to 70 => 30\n            assert_eq!(30.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_counter_increase_then_decrease_to_same_value() {\n        Spi::connect_mut(|client| {\n            increase_then_decrease_to_same_value(client);\n            let stmt = \"SELECT delta(counter_agg(ts, val)) FROM test\";\n            // In this case, counter goes 10, 30, 40 (reset + 10).\n            // Delta from 10 to 40 => 30\n            assert_eq!(30.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_counter_decrease() {\n        Spi::connect_mut(|client| {\n            decrease(client);\n            let stmt = \"SELECT idelta_left(counter_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_counter_increase() {\n        Spi::connect_mut(|client| {\n            increase(client);\n            let stmt = \"SELECT idelta_left(counter_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_counter_increase_then_decrease_to_same_value() {\n        Spi::connect_mut(|client| {\n            increase_then_decrease_to_same_value(client);\n            let stmt = \"SELECT idelta_left(counter_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_counter_decrease_then_increase_to_same_value() {\n        Spi::connect_mut(|client| {\n            decrease_then_increase_to_same_value(client);\n\n            let stmt = \"SELECT idelta_left(counter_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_counter_decrease() {\n        Spi::connect_mut(|client| {\n            decrease(client);\n            let stmt = \"SELECT idelta_right(counter_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_counter_increase() {\n        Spi::connect_mut(|client| {\n            increase(client);\n            let stmt = \"SELECT idelta_right(counter_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_counter_increase_then_decrease_to_same_value() {\n        Spi::connect_mut(|client| {\n            increase_then_decrease_to_same_value(client);\n            let stmt = \"SELECT idelta_right(counter_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_counter_decrease_then_increase_to_same_value() {\n        Spi::connect_mut(|client| {\n            decrease_then_increase_to_same_value(client);\n            let stmt = \"SELECT idelta_right(counter_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn counter_agg_interpolation() {\n        Spi::connect_mut(|client| {\n            client.update(\n                \"CREATE TABLE test(time timestamptz, value double precision, bucket timestamptz)\",\n                None,\n                &[]\n            ).unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                ('2020-1-1 10:00'::timestamptz, 10.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 12:00'::timestamptz, 40.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 16:00'::timestamptz, 20.0, '2020-1-1'::timestamptz),\n                ('2020-1-2 4:00'::timestamptz, 15.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 12:00'::timestamptz, 50.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 20:00'::timestamptz, 25.0, '2020-1-2'::timestamptz),\n                ('2020-1-3 4:00'::timestamptz, 30.0, '2020-1-3'::timestamptz),\n                ('2020-1-3 12:00'::timestamptz, 0.0, '2020-1-3'::timestamptz), \n                ('2020-1-3 16:00'::timestamptz, 35.0, '2020-1-3'::timestamptz)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut deltas = client\n                .update(\n                    r#\"SELECT\n                interpolated_delta(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, counter_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, start at 10, interpolated end of day is 10 (after reset), reset at 40 and 20\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(10. + 40. + 20. - 10.)\n            );\n            // Day 2, interpolated start is 10, interpolated end is 27.5, reset at 50\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(27.5 + 50. - 10.)\n            );\n            // Day 3, interpolated start is 27.5, end is 35, reset at 30\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(35. + 30. - 27.5)\n            );\n            assert!(deltas.next().is_none());\n\n            // test that the arrow version also returns the same result\n            let mut deltas = client\n                .update(\n                    r#\"SELECT\n                agg -> interpolated_delta(\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, counter_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, start at 10, interpolated end of day is 10 (after reset), reset at 40 and 20\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(10. + 40. + 20. - 10.)\n            );\n            // Day 2, interpolated start is 10, interpolated end is 27.5, reset at 50\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(27.5 + 50. - 10.)\n            );\n            // Day 3, interpolated start is 27.5, end is 35, reset at 30\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(35. + 30. - 27.5)\n            );\n            assert!(deltas.next().is_none());\n\n            let mut rates = client\n                .update(\n                    r#\"SELECT\n                interpolated_rate(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, counter_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, 14 hours (rate is per second)\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((10. + 40. + 20. - 10.) / (14. * 60. * 60.))\n            );\n            // Day 2, 24 hours\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((27.5 + 50. - 10.) / (24. * 60. * 60.))\n            );\n            // Day 3, 16 hours\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((35. + 30. - 27.5) / (16. * 60. * 60.))\n            );\n            assert!(rates.next().is_none());\n\n            // test that the arrow operator version also returns the same result\n            let mut rates = client\n                .update(\n                    r#\"SELECT\n                agg -> interpolated_rate(\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, counter_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, 14 hours (rate is per second)\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((10. + 40. + 20. - 10.) / (14. * 60. * 60.))\n            );\n            // Day 2, 24 hours\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((27.5 + 50. - 10.) / (24. * 60. * 60.))\n            );\n            // Day 3, 16 hours\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((35. + 30. - 27.5) / (16. * 60. * 60.))\n            );\n            assert!(rates.next().is_none());\n        });\n    }\n\n    #[pg_test]\n    fn interpolated_delta_with_aligned_point() {\n        Spi::connect_mut(|client| {\n            client.update(\n                \"CREATE TABLE test(time timestamptz, value double precision, bucket timestamptz)\",\n                None,\n                &[]\n            ).unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                ('2020-1-1 10:00'::timestamptz, 10.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 12:00'::timestamptz, 40.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 16:00'::timestamptz, 20.0, '2020-1-1'::timestamptz),\n                ('2020-1-2 0:00'::timestamptz, 15.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 12:00'::timestamptz, 50.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 20:00'::timestamptz, 25.0, '2020-1-2'::timestamptz)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut deltas = client\n                .update(\n                    r#\"SELECT\n                interpolated_delta(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, counter_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            // Day 1, start at 10, interpolated end of day is 15 (after reset), reset at 40 and 20\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(15. + 40. + 20. - 10.)\n            );\n            // Day 2, start is 15, end is 25, reset at 50\n            assert_eq!(\n                deltas.next().unwrap()[1].value().unwrap(),\n                Some(25. + 50. - 15.)\n            );\n            assert!(deltas.next().is_none());\n        });\n    }\n\n    #[pg_test]\n    fn irate_left_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       irate_left(counter_agg(ts, val)), \\\n                       counter_agg(ts, val) -> irate_left() \\\n                     FROM test\",\n                    f64\n                ),\n                0.16666666666666666,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn irate_right_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       irate_right(counter_agg(ts, val)), \\\n                       counter_agg(ts, val) -> irate_right() \\\n                     FROM test\",\n                    f64\n                ),\n                0.16666666666666666,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       idelta_left(counter_agg(ts, val)), \\\n                       counter_agg(ts, val) -> idelta_left() \\\n                     FROM test\",\n                    f64\n                ),\n                10.0,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       idelta_right(counter_agg(ts, val)), \\\n                       counter_agg(ts, val) -> idelta_right() \\\n                     FROM test\",\n                    f64\n                ),\n                10.0,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn num_resets_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       num_resets(counter_agg(ts, val))::float, \\\n                       (counter_agg(ts, val) -> num_resets())::float \\\n                     FROM test\",\n                    f64\n                ),\n                0.0,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn first_and_last_val() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_one!(\n                    client,\n                    \"SELECT \\\n                       first_val(counter_agg(ts, val)) \\\n                     FROM test\",\n                    f64\n                ),\n                10.0,\n            );\n\n            assert_relative_eq!(\n                select_one!(\n                    client,\n                    \"SELECT \\\n                       last_val(counter_agg(ts, val)) \\\n                     FROM test\",\n                    f64\n                ),\n                20.0,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn first_and_last_val_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       first_val(counter_agg(ts, val)), \\\n                       counter_agg(ts, val) -> first_val() \\\n                     FROM test\",\n                    f64\n                ),\n                10.0,\n            );\n\n            assert_relative_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       last_val(counter_agg(ts, val)), \\\n                       counter_agg(ts, val) -> last_val() \\\n                     FROM test\",\n                    f64\n                ),\n                20.0,\n            );\n        });\n    }\n\n    #[pg_test]\n    fn first_and_last_time() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n            client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n\n            assert_eq!(\n                select_one!(\n                    client,\n                    \"SELECT \\\n                       first_time(counter_agg(ts, val))::text \\\n                     FROM test\",\n                    &str\n                ),\n                \"2020-01-01 00:00:00+00\",\n            );\n\n            assert_eq!(\n                select_one!(\n                    client,\n                    \"SELECT \\\n                       last_time(counter_agg(ts, val))::text \\\n                     FROM test\",\n                    &str\n                ),\n                \"2020-01-01 00:01:00+00\",\n            );\n        });\n    }\n\n    #[pg_test]\n    fn first_and_last_time_arrow_match() {\n        Spi::connect_mut(|client| {\n            make_test_table(client, \"test\");\n            client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n\n            assert_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       first_time(counter_agg(ts, val))::text, \\\n                       (counter_agg(ts, val) -> first_time())::text \\\n                     FROM test\",\n                    &str\n                ),\n                \"2020-01-01 00:00:00+00\",\n            );\n\n            assert_eq!(\n                select_and_check_one!(\n                    client,\n                    \"SELECT \\\n                       last_time(counter_agg(ts, val))::text, \\\n                       (counter_agg(ts, val) -> last_time())::text \\\n                     FROM test\",\n                    &str\n                ),\n                \"2020-01-01 00:01:00+00\",\n            );\n        });\n    }\n\n    // #[pg_test]\n    // fn test_combine_aggregate(){\n    //     Spi::connect_mut(|client| {\n\n    //     });\n    // }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\npub(crate) mod testing {\n    pub fn decrease(client: &mut pgrx::spi::SpiClient) {\n        client\n            .update(\n                \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                None,\n                &[],\n            )\n            .unwrap();\n        client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n        client\n            .update(\n                r#\"INSERT INTO test VALUES\n                ('2020-01-01 00:00:00+00', 30.0),\n                ('2020-01-01 00:07:00+00', 10.0)\"#,\n                None,\n                &[],\n            )\n            .unwrap();\n    }\n\n    pub fn increase(client: &mut pgrx::spi::SpiClient) {\n        client\n            .update(\n                \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                None,\n                &[],\n            )\n            .unwrap();\n        client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n        client\n            .update(\n                r#\"INSERT INTO test VALUES\n                ('2020-01-01 00:00:00+00', 10.0),\n                ('2020-01-01 00:07:00+00', 30.0)\"#,\n                None,\n                &[],\n            )\n            .unwrap();\n    }\n\n    pub fn decrease_then_increase_to_same_value(client: &mut pgrx::spi::SpiClient) {\n        client\n            .update(\n                \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                None,\n                &[],\n            )\n            .unwrap();\n        client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n        client\n            .update(\n                r#\"INSERT INTO test VALUES\n                ('2020-01-01 00:00:00+00', 30.0),\n                ('2020-01-01 00:07:00+00', 10.0),\n                ('2020-01-01 00:08:00+00', 30.0)\"#,\n                None,\n                &[],\n            )\n            .unwrap();\n    }\n\n    pub fn increase_then_decrease_to_same_value(client: &mut pgrx::spi::SpiClient) {\n        client\n            .update(\n                \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                None,\n                &[],\n            )\n            .unwrap();\n        client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n        client\n            .update(\n                r#\"INSERT INTO test VALUES\n                ('2020-01-01 00:00:00+00', 10.0),\n                ('2020-01-01 00:07:00+00', 30.0),\n                ('2020-01-01 00:08:00+00', 10.0)\"#,\n                None,\n                &[],\n            )\n            .unwrap();\n    }\n\n    pub fn make_test_table(client: &mut pgrx::spi::SpiClient, name: &str) {\n        client\n            .update(\n                &format!(\"CREATE TABLE {name}(ts timestamptz, val DOUBLE PRECISION)\"),\n                None,\n                &[],\n            )\n            .unwrap();\n        client.update(\n                &format!(\"INSERT INTO {name} VALUES('2020-01-01 00:00:00+00', 10.0), ('2020-01-01 00:01:00+00', 20.0)\"),\n                None,\n                &[]\n            ).unwrap();\n    }\n}\n"
  },
  {
    "path": "extension/src/countminsketch.rs",
    "content": "use pgrx::*;\n\nuse aggregate_builder::aggregate;\nuse countminsketch::{CountMinHashFn, CountMinSketch as CountMinSketchInternal};\n\nuse crate::{\n    flatten,\n    palloc::{Inner, Internal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\n#[pg_schema]\npub mod toolkit_experimental {\n    use super::*;\n\n    pg_type! {\n        #[derive(Debug)]\n        struct CountMinSketch<'input> {\n            width: u32,\n            depth: u32,\n            counters: [i64; self.width * self.depth],\n        }\n    }\n\n    impl CountMinSketch<'_> {\n        fn new(width: u32, depth: u32, counters: Vec<i64>) -> Self {\n            let counters_arr = counters.into();\n            unsafe {\n                flatten!(CountMinSketch {\n                    width,\n                    depth,\n                    counters: counters_arr,\n                })\n            }\n        }\n\n        pub fn to_internal_countminsketch(&self) -> CountMinSketchInternal {\n            let depth: u64 = self.depth.into();\n            let hashfuncs = (1..=depth).map(CountMinHashFn::with_key).collect();\n\n            let mut counters: Vec<Vec<i64>> = Vec::with_capacity(self.depth as usize);\n            let row_width = self.width as usize;\n            for row in 0..self.depth {\n                let row_start = (row * self.width) as usize;\n                counters.push(\n                    self.counters\n                        .iter()\n                        .skip(row_start)\n                        .take(row_width)\n                        .collect(),\n                );\n            }\n\n            CountMinSketchInternal::new(\n                self.width as usize,\n                self.depth as usize,\n                hashfuncs,\n                counters,\n            )\n        }\n\n        pub fn from_internal_countminsketch(sketch: &mut CountMinSketchInternal) -> Self {\n            CountMinSketch::new(\n                sketch.width().try_into().unwrap(),\n                sketch.depth().try_into().unwrap(),\n                sketch.counters().iter().flatten().cloned().collect(),\n            )\n        }\n    }\n\n    ron_inout_funcs!(CountMinSketch<'input>);\n}\n\nuse toolkit_experimental::CountMinSketch;\n\n#[aggregate]\nimpl toolkit_experimental::count_min_sketch {\n    type State = CountMinSketchInternal;\n\n    fn transition(\n        state: Option<State>,\n        #[sql_type(\"text\")] value: Option<String>,\n        #[sql_type(\"float\")] error: f64,\n        #[sql_type(\"float\")] probability: f64,\n    ) -> Option<State> {\n        let value = match value {\n            None => return state,\n            Some(value) => value,\n        };\n\n        let mut state = match state {\n            None => CountMinSketchInternal::with_prob(error, probability),\n            Some(state) => state,\n        };\n\n        state.add_value(value);\n        Some(state)\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<CountMinSketch<'static>> {\n        state.map(CountMinSketch::from_internal_countminsketch)\n    }\n\n    const PARALLEL_SAFE: bool = true;\n\n    fn serialize(state: &mut State) -> bytea {\n        crate::do_serialize!(state)\n    }\n\n    fn deserialize(bytes: bytea) -> State {\n        crate::do_deserialize!(bytes, State)\n    }\n\n    fn combine(state1: Option<&State>, state2: Option<&State>) -> Option<State> {\n        match (state1, state2) {\n            (None, None) => None,\n            (None, Some(only)) | (Some(only), None) => Some(only.clone()),\n            (Some(a), Some(b)) => {\n                let (mut a, b) = (a.clone(), b.clone());\n                a.combine(b);\n                Some(a)\n            }\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn approx_count<'a>(item: String, aggregate: Option<CountMinSketch<'a>>) -> Option<i64> {\n    aggregate.map(|sketch| CountMinSketch::to_internal_countminsketch(&sketch).estimate(item))\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_countminsketch() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test (data TEXT)\", None, &[])\n                .unwrap();\n            client.update(\"INSERT INTO test SELECT generate_series(1, 100)::TEXT UNION ALL SELECT generate_series(1, 50)::TEXT\", None, &[]).unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(150), sanity);\n\n            client\n                .update(\n                    \"CREATE VIEW sketch AS \\\n                SELECT toolkit_experimental.count_min_sketch(data, 0.01, 0.01) \\\n                FROM test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM sketch\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert!(sanity.unwrap_or(0) > 0);\n\n            let (col1, col2, col3) = client\n                .update(\n                    \"SELECT \\\n                     toolkit_experimental.approx_count('1', count_min_sketch), \\\n                     toolkit_experimental.approx_count('51', count_min_sketch), \\\n                     toolkit_experimental.approx_count('101', count_min_sketch) \\\n                     FROM sketch\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_three::<i64, i64, i64>()\n                .unwrap();\n\n            // 0.01 => error param to the sketch, 150 => number of items added to the sketch\n            let err_margin = 0.01 * 150.0;\n\n            let items = [(col1, 2), (col2, 1), (col3, 0)];\n            for (approx_count, expected) in items {\n                let approx_count = approx_count.unwrap();\n                assert!(expected <= approx_count);\n\n                let upper_bound = err_margin + expected as f64;\n                let approx_count = approx_count as f64;\n                assert!(approx_count < upper_bound);\n            }\n        });\n    }\n\n    #[pg_test]\n    fn test_countminsketch_combine() {\n        Spi::connect_mut(|client| {\n            let combined = client\n                .update(\n\t\t    \"SELECT toolkit_experimental.approx_count('1', toolkit_experimental.count_min_sketch(v::text, 0.01, 0.01))\n                     FROM (SELECT * FROM generate_series(1, 100) v \\\n\t\t             UNION ALL \\\n                           SELECT * FROM generate_series(1, 100))  u(v)\",\n                    None,\n                    &[],\n                )\n                .unwrap().first()\n                .get_one::<i64>().unwrap();\n\n            let expected = 2;\n            // 0.01 => error param to the sketch, 200 => number of items added to the sketch\n            let err_margin = 0.01 * 200.0;\n\n            let approx_count = combined.unwrap();\n            assert!(expected <= approx_count);\n\n            let upper_bound = err_margin + expected as f64;\n            let approx_count = approx_count as f64;\n            assert!(approx_count < upper_bound);\n        });\n    }\n\n    #[pg_test]\n    fn countminsketch_io_test() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE io_test (value TEXT)\", None, &[])\n                .unwrap();\n            client.update(\"INSERT INTO io_test VALUES ('lorem'), ('ipsum'), ('dolor'), ('sit'), ('amet'), ('consectetur'), ('adipiscing'), ('elit')\", None, &[]).unwrap();\n\n            let sketch = client\n                .update(\n                    \"SELECT toolkit_experimental.count_min_sketch(value, 0.5, 0.01)::text FROM io_test\",\n                    None,\n                    &[]\n                )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                width:6,\\\n                depth:5,\\\n                counters:[\\\n                    1,2,2,1,1,1,\\\n                    0,0,2,3,1,2,\\\n                    1,0,3,0,4,0,\\\n                    1,3,2,0,1,1,\\\n                    0,0,4,3,0,1\\\n                    ]\\\n                )\";\n\n            assert_eq!(sketch, Some(expected.into()));\n        });\n    }\n\n    #[pg_test]\n    fn test_cms_null_input_yields_null_output() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\n                    \"SELECT toolkit_experimental.count_min_sketch(NULL::TEXT, 0.1, 0.1)::TEXT\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(output, None)\n        })\n    }\n\n    #[pg_test]\n    fn test_approx_count_null_input_yields_null_output() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\n                    \"SELECT toolkit_experimental.approx_count('1'::text, NULL::toolkit_experimental.countminsketch)\",\n                    None,\n                    &[]\n                )\n                .unwrap().first()\n                .get_one::<i64>().unwrap();\n            assert_eq!(output, None)\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/datum_utils.rs",
    "content": "use std::{\n    fmt,\n    hash::{BuildHasher, Hasher},\n    mem::size_of,\n    slice,\n};\n\nuse serde::{\n    de::{SeqAccess, Visitor},\n    ser::SerializeSeq,\n    Deserialize, Serialize,\n};\n\nuse pg_sys::{Datum, Oid};\nuse pgrx::*;\n\nuse crate::serialization::{PgCollationId, ShortTypeId};\n\npub(crate) unsafe fn deep_copy_datum(datum: Datum, typoid: Oid) -> Datum {\n    let tentry = pg_sys::lookup_type_cache(typoid, 0_i32);\n    if (*tentry).typbyval {\n        datum\n    } else if (*tentry).typlen > 0 {\n        // only varlena's can be toasted, manually copy anything with len >0\n        let size = (*tentry).typlen as usize;\n        let copy = pg_sys::palloc0(size);\n        std::ptr::copy(datum.cast_mut_ptr(), copy as *mut u8, size);\n        pg_sys::Datum::from(copy)\n    } else {\n        pg_sys::Datum::from(pg_sys::pg_detoast_datum_copy(datum.cast_mut_ptr()))\n    }\n}\n\n// If datum is an alloced type, free the associated memory\npub(crate) unsafe fn free_datum(datum: Datum, typoid: Oid) {\n    let tentry = pg_sys::lookup_type_cache(typoid, 0_i32);\n    if !(*tentry).typbyval {\n        pg_sys::pfree(datum.cast_mut_ptr())\n    }\n}\n\n// TODO: is there a better place for this?\n// Note that this requires an reference time to deal with variable length intervals (days or months)\npub fn ts_interval_sum_to_ms(\n    ref_time: &crate::raw::TimestampTz,\n    interval: &crate::raw::Interval,\n) -> i64 {\n    unsafe extern \"C-unwind\" {\n        #[allow(improper_ctypes)]\n        fn timestamptz_pl_interval(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n    }\n    let bound = unsafe {\n        pg_sys::DirectFunctionCall2Coll(\n            Some(timestamptz_pl_interval),\n            pg_sys::InvalidOid,\n            ref_time.0,\n            interval.0,\n        )\n    };\n    bound.value() as i64\n}\n\npub fn interval_to_ms(ref_time: &crate::raw::TimestampTz, interval: &crate::raw::Interval) -> i64 {\n    ts_interval_sum_to_ms(ref_time, interval) - ref_time.0.value() as i64\n}\n\npub struct TextSerializableDatumWriter {\n    flinfo: pg_sys::FmgrInfo,\n}\n\nimpl TextSerializableDatumWriter {\n    pub fn from_oid(typoid: Oid) -> Self {\n        let mut type_output = pg_sys::Oid::INVALID;\n        let mut typ_is_varlena = false;\n        let mut flinfo = unsafe { std::mem::MaybeUninit::zeroed().assume_init() };\n\n        unsafe {\n            pg_sys::getTypeOutputInfo(typoid, &mut type_output, &mut typ_is_varlena);\n            pg_sys::fmgr_info(type_output, &mut flinfo);\n        }\n\n        TextSerializableDatumWriter { flinfo }\n    }\n\n    pub fn make_serializable(&mut self, datum: Datum) -> TextSerializeableDatum {\n        TextSerializeableDatum(datum, &mut self.flinfo)\n    }\n}\n\npub struct DatumFromSerializedTextReader {\n    flinfo: pg_sys::FmgrInfo,\n    typ_io_param: pg_sys::Oid,\n}\n\nimpl DatumFromSerializedTextReader {\n    pub fn from_oid(typoid: Oid) -> Self {\n        let mut type_input = pg_sys::Oid::INVALID;\n        let mut typ_io_param = pg_sys::oids::Oid::INVALID;\n        let mut flinfo = unsafe { std::mem::MaybeUninit::zeroed().assume_init() };\n        unsafe {\n            pg_sys::getTypeInputInfo(typoid, &mut type_input, &mut typ_io_param);\n            pg_sys::fmgr_info(type_input, &mut flinfo);\n        }\n\n        DatumFromSerializedTextReader {\n            flinfo,\n            typ_io_param,\n        }\n    }\n\n    pub fn read_datum(&mut self, datum_str: &str) -> Datum {\n        let cstr = std::ffi::CString::new(datum_str).unwrap(); // TODO: error handling\n        let cstr_ptr = cstr.as_ptr() as *mut std::os::raw::c_char;\n        unsafe { pg_sys::InputFunctionCall(&mut self.flinfo, cstr_ptr, self.typ_io_param, -1) }\n    }\n}\n\npub struct TextSerializeableDatum(Datum, *mut pg_sys::FmgrInfo);\n\nimpl Serialize for TextSerializeableDatum {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        let chars = unsafe { pg_sys::OutputFunctionCall(self.1, self.0) };\n        let cstr = unsafe { std::ffi::CStr::from_ptr(chars) };\n        serializer.serialize_str(cstr.to_str().unwrap())\n    }\n}\n\npub(crate) struct DatumHashBuilder {\n    pub info: pg_sys::FunctionCallInfo,\n    pub type_id: pg_sys::Oid,\n    pub collation: pg_sys::Oid,\n}\n\nimpl DatumHashBuilder {\n    pub(crate) unsafe fn from_type_id(type_id: pg_sys::Oid, collation: Option<Oid>) -> Self {\n        let entry =\n            pg_sys::lookup_type_cache(type_id, pg_sys::TYPECACHE_HASH_EXTENDED_PROC_FINFO as _);\n        Self::from_type_cache_entry(entry, collation)\n    }\n\n    pub(crate) unsafe fn from_type_cache_entry(\n        tentry: *const pg_sys::TypeCacheEntry,\n        collation: Option<Oid>,\n    ) -> Self {\n        let flinfo = if (*tentry).hash_extended_proc_finfo.fn_addr.is_some() {\n            &(*tentry).hash_extended_proc_finfo\n        } else {\n            pgrx::error!(\"no hash function\");\n        };\n\n        // 1 argument for the key, 1 argument for the seed\n        let size =\n            size_of::<pg_sys::FunctionCallInfoBaseData>() + size_of::<pg_sys::NullableDatum>() * 2;\n        let info = pg_sys::palloc0(size) as pg_sys::FunctionCallInfo;\n\n        (*info).flinfo = flinfo as *const pg_sys::FmgrInfo as *mut pg_sys::FmgrInfo;\n        (*info).context = std::ptr::null_mut();\n        (*info).resultinfo = std::ptr::null_mut();\n        (*info).fncollation = (*tentry).typcollation;\n        (*info).isnull = false;\n        (*info).nargs = 1;\n\n        let collation = match collation {\n            Some(collation) => collation,\n            None => (*tentry).typcollation,\n        };\n\n        Self {\n            info,\n            type_id: (*tentry).type_id,\n            collation,\n        }\n    }\n}\n\nimpl Clone for DatumHashBuilder {\n    fn clone(&self) -> Self {\n        unsafe { DatumHashBuilder::from_type_id(self.type_id, Some(self.collation)) }\n    }\n}\n\nimpl BuildHasher for DatumHashBuilder {\n    type Hasher = DatumHashBuilder;\n\n    fn build_hasher(&self) -> Self::Hasher {\n        Self {\n            info: self.info,\n            type_id: self.type_id,\n            collation: self.collation,\n        }\n    }\n}\n\nimpl Hasher for DatumHashBuilder {\n    fn finish(&self) -> u64 {\n        //FIXME ehhh, this is wildly unsafe, should at least have a separate hash\n        //      buffer for each, probably should have separate args\n        let value = unsafe {\n            let value = (*(*self.info).flinfo).fn_addr.unwrap()(self.info);\n            (*self.info).args.as_mut_slice(1)[0] = pg_sys::NullableDatum {\n                value: Datum::from(0_usize),\n                isnull: true,\n            };\n            (*self.info).isnull = false;\n            //FIXME 32bit vs 64 bit get value from datum on 32b arch\n            value\n        };\n        value.value() as u64\n    }\n\n    fn write(&mut self, bytes: &[u8]) {\n        if bytes.len() != size_of::<usize>() {\n            panic!(\"invalid datum hash\")\n        }\n\n        let mut b = [0; size_of::<usize>()];\n        b[..size_of::<usize>()].clone_from_slice(&bytes[..size_of::<usize>()]);\n        self.write_usize(usize::from_ne_bytes(b))\n    }\n\n    fn write_usize(&mut self, i: usize) {\n        unsafe {\n            (*self.info).args.as_mut_slice(1)[0] = pg_sys::NullableDatum {\n                value: Datum::from(i),\n                isnull: false,\n            };\n            (*self.info).isnull = false;\n        }\n    }\n}\n\nimpl PartialEq for DatumHashBuilder {\n    fn eq(&self, other: &Self) -> bool {\n        self.type_id.eq(&other.type_id)\n    }\n}\n\nimpl Eq for DatumHashBuilder {}\n\nimpl Serialize for DatumHashBuilder {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        let collation = if self.collation == pg_sys::oids::Oid::INVALID {\n            None\n        } else {\n            Some(PgCollationId(self.collation))\n        };\n        (ShortTypeId(self.type_id), collation).serialize(serializer)\n    }\n}\n\nimpl<'de> Deserialize<'de> for DatumHashBuilder {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        let (type_id, collation) =\n            <(ShortTypeId, Option<PgCollationId>)>::deserialize(deserializer)?;\n        //FIXME no collation?\n        let deserialized = unsafe { Self::from_type_id(type_id.0, collation.map(|c| c.0)) };\n        Ok(deserialized)\n    }\n}\n\n#[inline]\nfn div_round_up(numerator: usize, divisor: usize) -> usize {\n    numerator.div_ceil(divisor)\n}\n\n#[inline]\nfn round_to_multiple(value: usize, multiple: usize) -> usize {\n    div_round_up(value, multiple) * multiple\n}\n\n#[inline]\nfn padded_va_len(ptr: *const pg_sys::varlena) -> usize {\n    unsafe { round_to_multiple(varsize_any(ptr), 8) }\n}\n\nflat_serialize_macro::flat_serialize! {\n    #[derive(Debug)]\n    struct DatumStore<'input> {\n        type_oid: crate::serialization::ShortTypeId,\n        data_len: u32,\n        // XXX this must be aligned to 8-bytes to ensure the stored data is correctly aligned\n        data: [u8; self.data_len],\n    }\n}\n\nimpl<'a> Serialize for DatumStore<'a> {\n    // TODO currently this always serializes inner data as text. When we start\n    // working on more-efficient network serialization, or we start using this\n    // in a transition state, we should use the binary format if we don't need\n    // human-readable output.\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        let mut writer = TextSerializableDatumWriter::from_oid(self.type_oid.0);\n        let count = self.iter().count();\n        let mut seq = serializer.serialize_seq(Some(count + 1))?;\n        seq.serialize_element(&self.type_oid.0.to_u32())?;\n        for element in self.iter() {\n            seq.serialize_element(&writer.make_serializable(element))?;\n        }\n        seq.end()\n    }\n}\n\nimpl<'a, 'de> Deserialize<'de> for DatumStore<'a> {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        struct DatumStoreVisitor<'a>(std::marker::PhantomData<core::cell::Cell<&'a ()>>);\n\n        impl<'de, 'a> Visitor<'de> for DatumStoreVisitor<'a> {\n            type Value = DatumStore<'a>;\n\n            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {\n                formatter.write_str(\"a sequence encoding a DatumStore object\")\n            }\n\n            fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n            where\n                A: SeqAccess<'de>,\n            {\n                let oid = Oid::from(seq.next_element::<u32>().unwrap().unwrap()); // TODO: error handling\n\n                // TODO separate human-readable and binary forms\n                let mut reader = DatumFromSerializedTextReader::from_oid(oid);\n\n                let mut data = vec![];\n                while let Some(next) = seq.next_element::<&str>()? {\n                    data.push(reader.read_datum(next));\n                }\n\n                Ok((oid, data).into())\n            }\n        }\n\n        deserializer.deserialize_seq(DatumStoreVisitor(std::marker::PhantomData))\n    }\n}\n\nimpl From<(Oid, Vec<Datum>)> for DatumStore<'_> {\n    fn from(input: (Oid, Vec<Datum>)) -> Self {\n        let (oid, datums) = input;\n        let (tlen, typbyval) = unsafe {\n            let tentry = pg_sys::lookup_type_cache(oid, 0_i32);\n            ((*tentry).typlen, (*tentry).typbyval)\n        };\n        assert!(tlen.is_positive() || tlen == -1 || tlen == -2);\n\n        if typbyval {\n            // Datum by value\n\n            // pad entries out to 8 byte aligned values...this may be a source of inefficiency\n            let data_len = round_to_multiple(tlen as usize, 8) as u32 * datums.len() as u32;\n\n            let mut data: Vec<u8> = vec![];\n            for datum in datums {\n                data.extend_from_slice(&datum.value().to_ne_bytes());\n            }\n\n            DatumStore {\n                type_oid: oid.into(),\n                data_len,\n                data: data.into(),\n            }\n        } else if tlen == -1 {\n            // Varlena\n\n            let mut ptrs = Vec::new();\n            let mut total_data_bytes = 0;\n\n            for datum in datums {\n                unsafe {\n                    let ptr =\n                        pg_sys::pg_detoast_datum_packed(datum.cast_mut_ptr::<pg_sys::varlena>());\n                    let va_len = varsize_any(ptr);\n\n                    ptrs.push(ptr);\n                    total_data_bytes += round_to_multiple(va_len, 8); // Round up to 8 byte boundary\n                }\n            }\n\n            let mut buffer = vec![0u8; total_data_bytes];\n\n            let mut target_byte = 0;\n            for ptr in ptrs {\n                unsafe {\n                    let va_len = varsize_any(ptr);\n                    std::ptr::copy(\n                        ptr as *const u8,\n                        std::ptr::addr_of_mut!(buffer[target_byte]),\n                        va_len,\n                    );\n                    target_byte += round_to_multiple(va_len, 8);\n                }\n            }\n\n            DatumStore {\n                type_oid: oid.into(),\n                data_len: total_data_bytes as u32,\n                data: buffer.into(),\n            }\n        } else if tlen == -2 {\n            // Null terminated string, should not be possible in this context\n            panic!(\"Unexpected null-terminated string type encountered.\");\n        } else {\n            // Fixed size reference\n\n            // Round size to multiple of 8 bytes\n            let len = round_to_multiple(tlen as usize, 8);\n            let total_length = len * datums.len();\n\n            let mut buffer = vec![0u8; total_length];\n            for (i, datum) in datums.iter().enumerate() {\n                unsafe {\n                    std::ptr::copy(\n                        datum.cast_mut_ptr(),\n                        std::ptr::addr_of_mut!(buffer[i * len]),\n                        tlen as usize,\n                    )\n                };\n            }\n\n            DatumStore {\n                type_oid: oid.into(),\n                data_len: total_length as u32,\n                data: buffer.into(),\n            }\n        }\n    }\n}\n\npub enum DatumStoreIterator<'a, 'b> {\n    Value {\n        iter: slice::Iter<'a, Datum>,\n    },\n    Varlena {\n        store: &'b DatumStore<'a>,\n        next_offset: u32,\n    },\n    FixedSize {\n        store: &'b DatumStore<'a>,\n        next_index: u32,\n        datum_size: u32,\n    },\n}\n\nimpl<'a, 'b> Iterator for DatumStoreIterator<'a, 'b> {\n    type Item = Datum;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        match self {\n            DatumStoreIterator::Value { iter } => iter.next().copied(),\n            DatumStoreIterator::Varlena { store, next_offset } => {\n                if *next_offset >= store.data_len {\n                    None\n                } else {\n                    unsafe {\n                        let va = store.data.slice().as_ptr().offset(*next_offset as _);\n                        *next_offset += padded_va_len(va as *const _) as u32;\n                        Some(pg_sys::Datum::from(va))\n                    }\n                }\n            }\n            DatumStoreIterator::FixedSize {\n                store,\n                next_index,\n                datum_size,\n            } => {\n                let idx = *next_index * *datum_size;\n                if idx >= store.data_len {\n                    None\n                } else {\n                    *next_index += 1;\n                    Some(pg_sys::Datum::from(unsafe {\n                        store.data.slice().as_ptr().offset(idx as _)\n                    }))\n                }\n            }\n        }\n    }\n}\n\nimpl<'a> DatumStore<'a> {\n    pub fn iter<'b>(&'b self) -> DatumStoreIterator<'a, 'b> {\n        unsafe {\n            let tentry = pg_sys::lookup_type_cache(self.type_oid.into(), 0_i32);\n            if (*tentry).typbyval {\n                // Datum by value\n                DatumStoreIterator::Value {\n                    // SAFETY `data` is guaranteed to be 8-byte aligned, so it should be safe to use as a slice\n                    iter: std::slice::from_raw_parts(\n                        self.data.as_slice().as_ptr() as *const Datum,\n                        self.data_len as usize / 8,\n                    )\n                    .iter(),\n                }\n            } else if (*tentry).typlen == -1 {\n                // Varlena\n                DatumStoreIterator::Varlena {\n                    store: self,\n                    next_offset: 0,\n                }\n            } else if (*tentry).typlen == -2 {\n                // Null terminated string\n                unreachable!()\n            } else {\n                // Fixed size reference\n                assert!((*tentry).typlen.is_positive());\n                DatumStoreIterator::FixedSize {\n                    store: self,\n                    next_index: 0,\n                    datum_size: round_to_multiple((*tentry).typlen as usize, 8) as u32,\n                }\n            }\n        }\n    }\n\n    pub fn into_anyelement_iter(self) -> impl Iterator<Item = AnyElement> + 'a {\n        let oid: pg_sys::Oid = self.type_oid.into();\n        self.into_iter()\n            .map(move |x| unsafe { AnyElement::from_polymorphic_datum(x, false, oid) }.unwrap())\n    }\n}\n\n// This is essentially the same as the DatumStoreIterator except that it takes ownership of the DatumStore,\n// there should be some way to efficiently merge these implementations\npub enum DatumStoreIntoIterator<'a> {\n    Value {\n        store: DatumStore<'a>,\n        next_idx: u32,\n    },\n    Varlena {\n        store: DatumStore<'a>,\n        next_offset: u32,\n    },\n    FixedSize {\n        store: DatumStore<'a>,\n        next_index: u32,\n        datum_size: u32,\n    },\n}\n\n// iterate over the set of values in the datum store\n// will return pointers into the datum store if it's a by-ref type\nimpl<'a> Iterator for DatumStoreIntoIterator<'a> {\n    type Item = Datum;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        match self {\n            DatumStoreIntoIterator::Value { store, next_idx } => {\n                let idx = *next_idx as usize;\n                let bound = store.data_len as usize / 8;\n                if idx >= bound {\n                    None\n                } else {\n                    // SAFETY `data` is guaranteed to be 8-byte aligned, so it is safe to use as a usize slice\n                    let dat = unsafe {\n                        std::slice::from_raw_parts(\n                            store.data.as_slice().as_ptr() as *const Datum,\n                            bound,\n                        )[idx]\n                    };\n                    *next_idx += 1;\n                    Some(dat)\n                }\n            }\n            DatumStoreIntoIterator::Varlena { store, next_offset } => {\n                if *next_offset >= store.data_len {\n                    None\n                } else {\n                    unsafe {\n                        let va = store.data.slice().as_ptr().offset(*next_offset as _);\n                        *next_offset += padded_va_len(va as *const _) as u32;\n                        Some(pg_sys::Datum::from(va))\n                    }\n                }\n            }\n            DatumStoreIntoIterator::FixedSize {\n                store,\n                next_index,\n                datum_size,\n            } => {\n                let idx = *next_index * *datum_size;\n                if idx >= store.data_len {\n                    None\n                } else {\n                    *next_index += 1;\n                    Some(pg_sys::Datum::from(unsafe {\n                        store.data.slice().as_ptr().offset(idx as _)\n                    }))\n                }\n            }\n        }\n    }\n}\n\nimpl<'a> IntoIterator for DatumStore<'a> {\n    type Item = Datum;\n    type IntoIter = DatumStoreIntoIterator<'a>;\n\n    fn into_iter(self) -> Self::IntoIter {\n        unsafe {\n            let tentry = pg_sys::lookup_type_cache(self.type_oid.into(), 0_i32);\n            if (*tentry).typbyval {\n                // Datum by value\n                DatumStoreIntoIterator::Value {\n                    store: self,\n                    next_idx: 0,\n                }\n            } else if (*tentry).typlen == -1 {\n                // Varlena\n                DatumStoreIntoIterator::Varlena {\n                    store: self,\n                    next_offset: 0,\n                }\n            } else if (*tentry).typlen == -2 {\n                // Null terminated string\n                unreachable!()\n            } else {\n                // Fixed size reference\n                assert!((*tentry).typlen.is_positive());\n                DatumStoreIntoIterator::FixedSize {\n                    store: self,\n                    next_index: 0,\n                    datum_size: round_to_multiple((*tentry).typlen as usize, 8) as u32,\n                }\n            }\n        }\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use crate::{build, palloc::Inner, pg_type, ron_inout_funcs};\n    use aggregate_builder::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_schema]\n    pub mod toolkit_experimental {\n        use super::*;\n        pg_type! {\n            #[derive(Debug)]\n            struct DatumStoreTester<'input> {\n                datums: DatumStore<'input>,\n            }\n        }\n        ron_inout_funcs!(DatumStoreTester<'input>);\n\n        #[aggregate]\n        impl toolkit_experimental::datum_test_agg {\n            type State = (Oid, Vec<Datum>);\n\n            fn transition(\n                state: Option<State>,\n                #[sql_type(\"AnyElement\")] value: AnyElement,\n            ) -> Option<State> {\n                match state {\n                    Some((oid, mut vector)) => {\n                        unsafe { vector.push(deep_copy_datum(value.datum(), oid)) };\n                        Some((oid, vector))\n                    }\n                    None => Some((\n                        value.oid(),\n                        vec![unsafe { deep_copy_datum(value.datum(), value.oid()) }],\n                    )),\n                }\n            }\n\n            fn finally(state: Option<&mut State>) -> Option<DatumStoreTester<'static>> {\n                state.map(|state| {\n                    build! {\n                        DatumStoreTester {\n                            datums: DatumStore::from(std::mem::take(state)),\n                        }\n                    }\n                })\n            }\n        }\n    }\n\n    #[pg_test]\n    fn test_value_datum_store() {\n        Spi::connect_mut(|client| {\n            let test = client.update(\"SELECT toolkit_experimental.datum_test_agg(r.data)::TEXT FROM (SELECT generate_series(10, 100, 10) as data) r\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap().unwrap();\n            let expected = \"(version:1,datums:[23,\\\"10\\\",\\\"20\\\",\\\"30\\\",\\\"40\\\",\\\"50\\\",\\\"60\\\",\\\"70\\\",\\\"80\\\",\\\"90\\\",\\\"100\\\"])\";\n            assert_eq!(test, expected);\n        });\n    }\n\n    #[pg_test]\n    fn test_varlena_datum_store() {\n        Spi::connect_mut(|client| {\n            let test = client.update(\"SELECT toolkit_experimental.datum_test_agg(r.data)::TEXT FROM (SELECT generate_series(10, 100, 10)::TEXT as data) r\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap().unwrap();\n            let expected = \"(version:1,datums:[25,\\\"10\\\",\\\"20\\\",\\\"30\\\",\\\"40\\\",\\\"50\\\",\\\"60\\\",\\\"70\\\",\\\"80\\\",\\\"90\\\",\\\"100\\\"])\";\n            assert_eq!(test, expected);\n        });\n    }\n\n    #[pg_test]\n    fn test_byref_datum_store() {\n        Spi::connect_mut(|client| {\n            let test = client.update(\"SELECT toolkit_experimental.datum_test_agg(r.data)::TEXT FROM (SELECT (generate_series(10, 100, 10)::TEXT || ' seconds')::INTERVAL as data) r\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap().unwrap();\n            let expected = \"(version:1,datums:[1186,\\\"00:00:10\\\",\\\"00:00:20\\\",\\\"00:00:30\\\",\\\"00:00:40\\\",\\\"00:00:50\\\",\\\"00:01:00\\\",\\\"00:01:10\\\",\\\"00:01:20\\\",\\\"00:01:30\\\",\\\"00:01:40\\\"])\";\n            assert_eq!(test, expected);\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/duration.rs",
    "content": "//! Utilities for working with durations. Parsing of duration units is intended to match how\n//! PostgreSQL parses duration units. Currently units longer than an hour are unsupported since\n//! the length of days varies when in a timezone with daylight savings time.\n\nuse core::fmt::{self, Formatter};\n\n// Canonical PostgreSQL units: https://github.com/postgres/postgres/blob/b76fb6c2a99eb7d49f96e56599fef1ffc1c134c9/src/include/utils/datetime.h#L48-L60\n#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord)]\npub enum DurationUnit {\n    // units should be ordered smallest -> largest\n    Microsec,\n    Millisec,\n    Second,\n    Minute,\n    Hour,\n}\n\nimpl DurationUnit {\n    pub fn microseconds(self) -> u32 {\n        match self {\n            Self::Microsec => 1,\n            Self::Millisec => 1000,\n            Self::Second => 1_000_000,\n            Self::Minute => 60_000_000,\n            Self::Hour => 3_600_000_000,\n        }\n    }\n\n    /// Convert `amount` of a unit to another unit.\n    pub fn convert_unit(self, amount: f64, to: Self) -> f64 {\n        let microseconds = amount * (self.microseconds() as f64);\n        microseconds / (to.microseconds() as f64)\n    }\n\n    /// Tries to get a duration unit from a string, returning `None` if no known unit matched.\n    pub fn from_str(s: &str) -> Option<Self> {\n        // Aliases for canonical units: https://github.com/postgres/postgres/blob/b76fb6c2a99eb7d49f96e56599fef1ffc1c134c9/src/backend/utils/adt/datetime.c#L187-L247\n        match s.to_lowercase().as_str() {\n            \"usecond\" | \"microsecond\" | \"microseconds\" | \"microsecon\" | \"us\" | \"usec\"\n            | \"useconds\" | \"usecs\" => Some(Self::Microsec),\n            \"msecond\" | \"millisecond\" | \"milliseconds\" | \"millisecon\" | \"ms\" | \"msec\"\n            | \"mseconds\" | \"msecs\" => Some(Self::Millisec),\n            \"second\" | \"s\" | \"sec\" | \"seconds\" | \"secs\" => Some(Self::Second),\n            \"minute\" | \"m\" | \"min\" | \"mins\" | \"minutes\" => Some(Self::Minute),\n            \"hour\" | \"hours\" | \"h\" | \"hr\" | \"hrs\" => Some(Self::Hour),\n            _ => None,\n        }\n    }\n}\n\nimpl fmt::Display for DurationUnit {\n    fn fmt(&self, f: &mut Formatter) -> fmt::Result {\n        match self {\n            DurationUnit::Microsec => write!(f, \"microsecond\"),\n            DurationUnit::Millisec => write!(f, \"millisecond\"),\n            DurationUnit::Second => write!(f, \"second\"),\n            DurationUnit::Minute => write!(f, \"minute\"),\n            DurationUnit::Hour => write!(f, \"hour\"),\n        }\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn convert_unit() {\n        let load_time_secs = 75.0;\n        let load_time_mins =\n            DurationUnit::convert_unit(DurationUnit::Second, load_time_secs, DurationUnit::Minute);\n        assert_eq!(load_time_mins, 1.25);\n    }\n\n    #[test]\n    fn parse_unit() {\n        assert_eq!(\n            DurationUnit::from_str(\"usecs\"),\n            Some(DurationUnit::Microsec)\n        );\n        assert_eq!(DurationUnit::from_str(\"MINUTE\"), Some(DurationUnit::Minute));\n        assert_eq!(\n            DurationUnit::from_str(\"MiLlIsEcOn\"),\n            Some(DurationUnit::Millisec)\n        );\n        assert_eq!(DurationUnit::from_str(\"pahar\"), None);\n        assert_eq!(DurationUnit::from_str(\"\"), None);\n    }\n}\n"
  },
  {
    "path": "extension/src/frequency.rs",
    "content": "//! Based on the paper: https://cs.ucsb.edu/sites/default/files/documents/2005-23.pdf\n\nuse std::fmt;\n\nuse pgrx::{\n    iter::{SetOfIterator, TableIterator},\n    *,\n};\n\nuse pg_sys::{Datum, Oid};\n\nuse serde::{\n    de::{SeqAccess, Visitor},\n    ser::SerializeSeq,\n    Deserialize, Serialize,\n};\n\nuse crate::{\n    accessors::{\n        AccessorIntoValues, AccessorMaxFrequencyInt, AccessorMinFrequencyInt, AccessorTopNCount,\n        AccessorTopn,\n    },\n    aggregate_utils::{get_collation_or_default, in_aggregate_context},\n    build,\n    datum_utils::{\n        deep_copy_datum, DatumFromSerializedTextReader, DatumHashBuilder, DatumStore,\n        TextSerializableDatumWriter,\n    },\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_any_element::{PgAnyElement, PgAnyElementHashMap},\n    pg_type,\n    raw::{bytea, text},\n    ron_inout_funcs,\n};\n\nuse spfunc::zeta::zeta;\nuse statrs::function::harmonic::gen_harmonic;\n\n// Helper functions for zeta distribution\n\n// Default s-value\nconst DEFAULT_ZETA_SKEW: f64 = 1.1;\n\n// probability of the nth element of a zeta distribution\nfn zeta_eq_n(skew: f64, n: u64) -> f64 {\n    1.0 / zeta(skew) * (n as f64).powf(-skew)\n}\n// cumulative distribution <= n in a zeta distribution\nfn zeta_le_n(skew: f64, n: u64) -> f64 {\n    gen_harmonic(n, skew) / zeta(skew)\n}\n\nstruct SpaceSavingEntry {\n    value: Datum,\n    count: u64,\n    overcount: u64,\n}\n\nimpl SpaceSavingEntry {\n    fn clone(&self, typoid: Oid) -> SpaceSavingEntry {\n        SpaceSavingEntry {\n            value: unsafe { deep_copy_datum(self.value, typoid) },\n            count: self.count,\n            overcount: self.overcount,\n        }\n    }\n}\n\npub struct SpaceSavingTransState {\n    entries: Vec<SpaceSavingEntry>,\n    indices: PgAnyElementHashMap<usize>,\n    total_vals: u64,\n    freq_param: f64, // This is the minimum frequency for a freq_agg or the skew for a mcv_agg\n    topn: u32,       // 0 for freq_agg, creation parameter for mcv_agg\n    max_size: u32,   // Maximum size for indices\n}\n\nimpl Clone for SpaceSavingTransState {\n    fn clone(&self) -> Self {\n        let mut new_state = Self {\n            entries: vec![],\n            indices: PgAnyElementHashMap::with_hasher(self.indices.hasher().clone()),\n            total_vals: self.total_vals,\n            freq_param: self.freq_param,\n            max_size: self.max_size,\n            topn: self.topn,\n        };\n\n        let typoid = self.type_oid();\n        for entry in &self.entries {\n            new_state.entries.push(SpaceSavingEntry {\n                value: unsafe { deep_copy_datum(entry.value, typoid) },\n                count: entry.count,\n                overcount: entry.overcount,\n            })\n        }\n        new_state.update_all_map_indices();\n        new_state\n    }\n}\n\n// SpaceSavingTransState is a little tricky to serialize due to needing the typ oid to serialize the Datums.\n// This sort of requirement doesn't play nicely with the serde framework, so as a workaround we simply\n// serialize the object as one big sequence.  The serialized sequence should look like this:\n//   total_vals as u64\n//   min_freq as f64\n//   max_idx as u32\n//   topn as u32\n//   indices.hasher as DatumHashBuilder\n//   entries as repeated (str, u64, u64) tuples\nimpl Serialize for SpaceSavingTransState {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        let mut seq = serializer.serialize_seq(Some(self.entries.len() + 5))?;\n        seq.serialize_element(&self.total_vals)?;\n        seq.serialize_element(&self.freq_param)?;\n        seq.serialize_element(&self.max_size)?;\n        seq.serialize_element(&self.topn)?;\n        seq.serialize_element(&self.indices.hasher())?;\n\n        // TODO JOSH use a writer that switches based on whether we want binary or not\n        let mut writer = TextSerializableDatumWriter::from_oid(self.type_oid());\n\n        for entry in &self.entries {\n            seq.serialize_element(&(\n                writer.make_serializable(entry.value),\n                entry.count,\n                entry.overcount,\n            ))?;\n        }\n        seq.end()\n    }\n}\n\nimpl<'de> Deserialize<'de> for SpaceSavingTransState {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        struct FrequencyTransStateVisitor();\n\n        impl<'de> Visitor<'de> for FrequencyTransStateVisitor {\n            type Value = SpaceSavingTransState;\n\n            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {\n                formatter.write_str(\"a sequence encoding a FrequencyTransState object\")\n            }\n\n            fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n            where\n                A: SeqAccess<'de>,\n            {\n                let total_vals = seq.next_element::<u64>()?.unwrap();\n                let min_freq = seq.next_element::<f64>()?.unwrap();\n                let max_size = seq.next_element::<u32>()?.unwrap();\n                let topn = seq.next_element::<u32>()?.unwrap();\n                let hasher = seq.next_element::<DatumHashBuilder>()?.unwrap();\n\n                let mut state = SpaceSavingTransState {\n                    entries: vec![],\n                    indices: PgAnyElementHashMap::with_hasher(hasher),\n                    total_vals,\n                    freq_param: min_freq,\n                    max_size,\n                    topn,\n                };\n\n                let typid = state.type_oid();\n                let mut reader = DatumFromSerializedTextReader::from_oid(typid);\n\n                while let Some((datum_str, count, overcount)) =\n                    seq.next_element::<(&str, u64, u64)>()?\n                {\n                    let datum = reader.read_datum(datum_str);\n\n                    state.entries.push(SpaceSavingEntry {\n                        value: unsafe { deep_copy_datum(datum, typid) },\n                        count,\n                        overcount,\n                    });\n                }\n                state.update_all_map_indices();\n                Ok(state)\n            }\n        }\n\n        deserializer.deserialize_seq(FrequencyTransStateVisitor())\n    }\n}\n\nimpl SpaceSavingTransState {\n    fn max_size_for_freq(min_freq: f64) -> u32 {\n        (1. / min_freq) as u32 + 1\n    }\n\n    fn freq_agg_from_type_id(min_freq: f64, typ: pg_sys::Oid, collation: Option<Oid>) -> Self {\n        SpaceSavingTransState {\n            entries: vec![],\n            indices: PgAnyElementHashMap::new(typ, collation),\n            total_vals: 0,\n            freq_param: min_freq,\n            max_size: SpaceSavingTransState::max_size_for_freq(min_freq),\n            topn: 0,\n        }\n    }\n\n    fn mcv_agg_from_type_id(\n        skew: f64,\n        nval: u32,\n        typ: pg_sys::Oid,\n        collation: Option<Oid>,\n    ) -> Self {\n        if nval == 0 {\n            pgrx::error!(\"mcv aggregate requires an n value > 0\")\n        }\n        if skew <= 1.0 {\n            pgrx::error!(\"mcv aggregate requires a skew factor > 1.0\")\n        }\n\n        let prob_eq_n = zeta_eq_n(skew, nval as u64);\n        let prob_lt_n = zeta_le_n(skew, nval as u64 - 1);\n\n        SpaceSavingTransState {\n            entries: vec![],\n            indices: PgAnyElementHashMap::new(typ, collation),\n            total_vals: 0,\n            freq_param: skew,\n            max_size: nval - 1\n                + SpaceSavingTransState::max_size_for_freq(prob_eq_n / (1.0 - prob_lt_n)),\n            topn: nval,\n        }\n    }\n\n    fn ingest_aggregate_data(\n        &mut self,\n        val_count: u64,\n        values: &DatumStore,\n        counts: &[u64],\n        overcounts: &[u64],\n    ) {\n        assert_eq!(self.total_vals, 0); // This should only be called on an empty aggregate\n        self.total_vals = val_count;\n\n        for (idx, datum) in values.iter().enumerate() {\n            self.entries.push(SpaceSavingEntry {\n                value: unsafe { deep_copy_datum(datum, self.indices.typoid()) },\n                count: counts[idx],\n                overcount: overcounts[idx],\n            });\n            self.indices\n                .insert((self.entries[idx].value, self.type_oid()).into(), idx);\n        }\n    }\n\n    fn ingest_aggregate_ints(\n        &mut self,\n        val_count: u64,\n        values: &[i64],\n        counts: &[u64],\n        overcounts: &[u64],\n    ) {\n        assert_eq!(self.total_vals, 0); // This should only be called on an empty aggregate\n        assert_eq!(self.type_oid(), pg_sys::INT8OID);\n        self.total_vals = val_count;\n\n        for (idx, val) in values.iter().enumerate() {\n            self.entries.push(SpaceSavingEntry {\n                value: Datum::from(*val),\n                count: counts[idx],\n                overcount: overcounts[idx],\n            });\n            self.indices\n                .insert((self.entries[idx].value, self.type_oid()).into(), idx);\n        }\n    }\n\n    fn type_oid(&self) -> Oid {\n        self.indices.typoid()\n    }\n\n    fn add(&mut self, element: PgAnyElement) {\n        self.total_vals += 1;\n        if let Some(idx) = self.indices.get(&element) {\n            let idx = *idx;\n            self.entries[idx].count += 1;\n            self.move_left(idx);\n        } else if self.entries.len() < self.max_size as usize {\n            let new_idx = self.entries.len();\n            self.entries.push(SpaceSavingEntry {\n                value: element.deep_copy_datum(),\n                count: 1,\n                overcount: 0,\n            });\n\n            // Important to create the indices entry using the datum in the local context\n            self.indices.insert(\n                (self.entries[new_idx].value, self.type_oid()).into(),\n                new_idx,\n            );\n        } else {\n            let new_value = element.deep_copy_datum();\n\n            // TODO: might be more efficient to replace the lowest indexed tail value (count matching last) and not call move_up\n            let typoid = self.type_oid();\n            let entry = self.entries.last_mut().unwrap();\n            self.indices.remove(&(entry.value, typoid).into());\n            entry.value = new_value; // JOSH FIXME should we pfree() old value if by-ref?\n            entry.overcount = entry.count;\n            entry.count += 1;\n            self.indices\n                .insert((new_value, typoid).into(), self.entries.len() - 1);\n            self.move_left(self.entries.len() - 1);\n        }\n    }\n\n    // swap element i with an earlier element in the 'entries' vector to maintain decreasing order\n    fn move_left(&mut self, i: usize) {\n        let count = self.entries[i].count;\n        let mut target = i;\n        while target > 0 && self.entries[target - 1].count < count {\n            target -= 1;\n        }\n        if target != i {\n            self.entries.swap(i, target);\n\n            self.update_map_index(i);\n            self.update_map_index(target);\n        }\n    }\n\n    // Adds the 'indices' lookup entry for the value at 'entries' index i\n    fn update_map_index(&mut self, i: usize) {\n        let element_for_i = (self.entries[i].value, self.type_oid()).into();\n        if let Some(entry) = self.indices.get_mut(&element_for_i) {\n            *entry = i;\n        } else {\n            self.indices.insert(element_for_i, i);\n        }\n    }\n\n    fn update_all_map_indices(&mut self) {\n        for i in 0..self.entries.len() {\n            self.update_map_index(i);\n        }\n    }\n\n    fn combine(one: &SpaceSavingTransState, two: &SpaceSavingTransState) -> SpaceSavingTransState {\n        // This takes an entry from a TransState, updates it with any state from the other TransState, and adds the result into the map\n        fn new_entry(\n            entry: &SpaceSavingEntry,\n            other: &SpaceSavingTransState,\n            map: &mut PgAnyElementHashMap<SpaceSavingEntry>,\n        ) {\n            let typoid = other.type_oid();\n\n            let mut new_ent = entry.clone(typoid);\n            let new_dat = (new_ent.value, typoid).into();\n            match other.indices.get(&new_dat) {\n                Some(&idx) => {\n                    new_ent.count += other.entries[idx].count;\n                    new_ent.overcount += other.entries[idx].overcount;\n                }\n                None => {\n                    // If the entry value isn't present in the other state, we have to assume that it was recently bumped (unless the other state is not fully populated).\n                    let min = if other.indices.len() < other.max_size as usize {\n                        0\n                    } else {\n                        other.entries.last().unwrap().count\n                    };\n                    new_ent.count += min;\n                    new_ent.overcount += min;\n                }\n            }\n            map.insert(new_dat, new_ent);\n        }\n\n        let hasher = one.indices.hasher().clone();\n        let mut temp = PgAnyElementHashMap::with_hasher(hasher);\n\n        // First go through the first state, and add all entries (updated with other other state) to our temporary hashmap\n        for entry in &one.entries {\n            new_entry(entry, two, &mut temp);\n        }\n\n        // Next add in anything in the second state that isn't already in the map.\n        // TODO JOSH does filter make this easier to read\n        for entry in &two.entries {\n            if !temp.contains_key(&(entry.value, one.type_oid()).into()) {\n                new_entry(entry, one, &mut temp);\n            }\n        }\n\n        // TODO: get this into_iter working without making temp.0 public\n        let mut entries: Vec<SpaceSavingEntry> = temp.0.into_values().collect();\n        entries.sort_by(|a, b| b.count.partial_cmp(&a.count).unwrap()); // swap a and b for descending\n\n        entries.truncate(one.max_size as usize);\n\n        let mut result = SpaceSavingTransState {\n            entries,\n            indices: PgAnyElementHashMap::with_hasher(one.indices.hasher().clone()),\n            total_vals: one.total_vals + two.total_vals,\n            freq_param: one.freq_param,\n            max_size: one.max_size,\n            topn: one.topn,\n        };\n\n        result.update_all_map_indices();\n        result\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct SpaceSavingAggregate<'input> {\n        type_oid: u32,\n        num_values: u32,\n        values_seen: u64,\n        freq_param: f64,\n        topn: u64, // bump this up to u64 to keep alignment\n        counts: [u64; self.num_values], // JOSH TODO look at AoS instead of SoA at some point\n        overcounts: [u64; self.num_values],\n        datums: DatumStore<'input>,\n    }\n}\n\nimpl<'input> From<&SpaceSavingTransState> for SpaceSavingAggregate<'input> {\n    fn from(trans: &SpaceSavingTransState) -> Self {\n        let mut values = Vec::new();\n        let mut counts = Vec::new();\n        let mut overcounts = Vec::new();\n\n        for entry in &trans.entries {\n            values.push(entry.value);\n            counts.push(entry.count);\n            overcounts.push(entry.overcount);\n        }\n\n        build! {\n            SpaceSavingAggregate {\n                type_oid: trans.type_oid().into(),\n                num_values: trans.entries.len() as _,\n                values_seen: trans.total_vals,\n                freq_param: trans.freq_param,\n                topn: trans.topn as u64,\n                counts: counts.into(),\n                overcounts: overcounts.into(),\n                datums: DatumStore::from((trans.type_oid(), values)),\n            }\n        }\n    }\n}\n\nimpl<'input> From<(&SpaceSavingAggregate<'input>, &pg_sys::FunctionCallInfo)>\n    for SpaceSavingTransState\n{\n    fn from(data_in: (&SpaceSavingAggregate<'input>, &pg_sys::FunctionCallInfo)) -> Self {\n        let (agg, fcinfo) = data_in;\n        let collation = get_collation_or_default(*fcinfo);\n        let mut trans = if agg.topn == 0 {\n            SpaceSavingTransState::freq_agg_from_type_id(\n                agg.freq_param,\n                Oid::from(agg.type_oid),\n                collation,\n            )\n        } else {\n            SpaceSavingTransState::mcv_agg_from_type_id(\n                agg.freq_param,\n                agg.topn as u32,\n                Oid::from(agg.type_oid),\n                collation,\n            )\n        };\n        trans.ingest_aggregate_data(\n            agg.values_seen,\n            &agg.datums,\n            agg.counts.as_slice(),\n            agg.overcounts.as_slice(),\n        );\n        trans\n    }\n}\n\nron_inout_funcs!(SpaceSavingAggregate<'input>);\n\npg_type! {\n    #[derive(Debug)]\n    struct SpaceSavingBigIntAggregate<'input> {\n        num_values: u32,\n        topn: u32,\n        values_seen: u64,\n        freq_param: f64,\n        counts: [u64; self.num_values], // JOSH TODO look at AoS instead of SoA at some point\n        overcounts: [u64; self.num_values],\n        datums: [i64; self.num_values],\n    }\n}\n\nimpl<'input> From<&SpaceSavingTransState> for SpaceSavingBigIntAggregate<'input> {\n    fn from(trans: &SpaceSavingTransState) -> Self {\n        assert_eq!(trans.type_oid(), pg_sys::INT8OID);\n\n        let mut values = Vec::new();\n        let mut counts = Vec::new();\n        let mut overcounts = Vec::new();\n\n        for entry in &trans.entries {\n            values.push(entry.value.value() as i64);\n            counts.push(entry.count);\n            overcounts.push(entry.overcount);\n        }\n\n        build! {\n            SpaceSavingBigIntAggregate {\n                num_values: trans.entries.len() as _,\n                values_seen: trans.total_vals,\n                freq_param: trans.freq_param,\n                topn: trans.topn,\n                counts: counts.into(),\n                overcounts: overcounts.into(),\n                datums: values.into(),\n            }\n        }\n    }\n}\n\nimpl<'input>\n    From<(\n        &SpaceSavingBigIntAggregate<'input>,\n        &pg_sys::FunctionCallInfo,\n    )> for SpaceSavingTransState\n{\n    fn from(\n        data_in: (\n            &SpaceSavingBigIntAggregate<'input>,\n            &pg_sys::FunctionCallInfo,\n        ),\n    ) -> Self {\n        let (agg, fcinfo) = data_in;\n        let collation = get_collation_or_default(*fcinfo);\n        let mut trans = if agg.topn == 0 {\n            SpaceSavingTransState::freq_agg_from_type_id(agg.freq_param, pg_sys::INT8OID, collation)\n        } else {\n            SpaceSavingTransState::mcv_agg_from_type_id(\n                agg.freq_param,\n                agg.topn,\n                pg_sys::INT8OID,\n                collation,\n            )\n        };\n        trans.ingest_aggregate_ints(\n            agg.values_seen,\n            agg.datums.as_slice(),\n            agg.counts.as_slice(),\n            agg.overcounts.as_slice(),\n        );\n        trans\n    }\n}\n\nron_inout_funcs!(SpaceSavingBigIntAggregate<'input>);\n\npg_type! {\n    #[derive(Debug)]\n    struct SpaceSavingTextAggregate<'input> {\n        num_values: u32,\n        topn: u32,\n        values_seen: u64,\n        freq_param: f64,\n        counts: [u64; self.num_values], // JOSH TODO look at AoS instead of SoA at some point\n        overcounts: [u64; self.num_values],\n        datums: DatumStore<'input>,\n    }\n}\n\nimpl<'input> From<&SpaceSavingTransState> for SpaceSavingTextAggregate<'input> {\n    fn from(trans: &SpaceSavingTransState) -> Self {\n        assert_eq!(trans.type_oid(), pg_sys::TEXTOID);\n\n        let mut values = Vec::new();\n        let mut counts = Vec::new();\n        let mut overcounts = Vec::new();\n\n        for entry in &trans.entries {\n            values.push(entry.value);\n            counts.push(entry.count);\n            overcounts.push(entry.overcount);\n        }\n\n        build! {\n            SpaceSavingTextAggregate {\n                num_values: trans.entries.len() as _,\n                values_seen: trans.total_vals,\n                freq_param: trans.freq_param,\n                topn: trans.topn,\n                counts: counts.into(),\n                overcounts: overcounts.into(),\n                datums: DatumStore::from((trans.type_oid(), values)),\n            }\n        }\n    }\n}\n\nimpl<'input> From<(&SpaceSavingTextAggregate<'input>, &pg_sys::FunctionCallInfo)>\n    for SpaceSavingTransState\n{\n    fn from(data_in: (&SpaceSavingTextAggregate<'input>, &pg_sys::FunctionCallInfo)) -> Self {\n        let (agg, fcinfo) = data_in;\n        let collation = get_collation_or_default(*fcinfo);\n        let mut trans = if agg.topn == 0 {\n            SpaceSavingTransState::freq_agg_from_type_id(agg.freq_param, pg_sys::TEXTOID, collation)\n        } else {\n            SpaceSavingTransState::mcv_agg_from_type_id(\n                agg.freq_param,\n                agg.topn,\n                pg_sys::TEXTOID,\n                collation,\n            )\n        };\n        trans.ingest_aggregate_data(\n            agg.values_seen,\n            &agg.datums,\n            agg.counts.as_slice(),\n            agg.overcounts.as_slice(),\n        );\n        trans\n    }\n}\n\nron_inout_funcs!(SpaceSavingTextAggregate<'input>);\n\n#[pg_extern(immutable, parallel_safe)]\npub fn mcv_agg_trans(\n    state: Internal,\n    n: i32,\n    value: Option<AnyElement>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    mcv_agg_with_skew_trans(state, n, DEFAULT_ZETA_SKEW, value, fcinfo)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn mcv_agg_bigint_trans(\n    state: Internal,\n    n: i32,\n    value: Option<i64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    mcv_agg_with_skew_bigint_trans(state, n, DEFAULT_ZETA_SKEW, value, fcinfo)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn mcv_agg_text_trans(\n    state: Internal,\n    n: i32,\n    value: Option<crate::raw::text>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    mcv_agg_with_skew_text_trans(state, n, DEFAULT_ZETA_SKEW, value, fcinfo)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn mcv_agg_with_skew_trans(\n    state: Internal,\n    n: i32,\n    skew: f64,\n    value: Option<AnyElement>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    space_saving_trans(\n        unsafe { state.to_inner() },\n        value,\n        fcinfo,\n        |typ, collation| {\n            SpaceSavingTransState::mcv_agg_from_type_id(skew, n as u32, typ, collation)\n        },\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn mcv_agg_with_skew_bigint_trans(\n    state: Internal,\n    n: i32,\n    skew: f64,\n    value: Option<i64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let value = match value {\n        None => None,\n        Some(val) => unsafe {\n            AnyElement::from_polymorphic_datum(pg_sys::Datum::from(val), false, pg_sys::INT8OID)\n        },\n    };\n\n    space_saving_trans(\n        unsafe { state.to_inner() },\n        value,\n        fcinfo,\n        |typ, collation| {\n            SpaceSavingTransState::mcv_agg_from_type_id(skew, n as u32, typ, collation)\n        },\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn mcv_agg_with_skew_text_trans(\n    state: Internal,\n    n: i32,\n    skew: f64,\n    value: Option<crate::raw::text>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let txt = value.map(|v| unsafe { pg_sys::pg_detoast_datum_copy(v.0.cast_mut_ptr()) });\n    let value = match txt {\n        None => None,\n        Some(val) => unsafe {\n            AnyElement::from_polymorphic_datum(pg_sys::Datum::from(val), false, pg_sys::TEXTOID)\n        },\n    };\n\n    space_saving_trans(\n        unsafe { state.to_inner() },\n        value,\n        fcinfo,\n        |typ, collation| {\n            SpaceSavingTransState::mcv_agg_from_type_id(skew, n as u32, typ, collation)\n        },\n    )\n    .internal()\n}\n\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\npub fn freq_agg_trans(\n    state: Internal,\n    freq: f64,\n    value: Option<AnyElement>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    if freq <= 0. || freq >= 1.0 {\n        pgrx::error!(\"frequency aggregate requires a frequency in the range (0.0, 1.0)\")\n    }\n\n    space_saving_trans(\n        unsafe { state.to_inner() },\n        value,\n        fcinfo,\n        |typ, collation| SpaceSavingTransState::freq_agg_from_type_id(freq, typ, collation),\n    )\n    .internal()\n}\n\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\npub fn freq_agg_bigint_trans(\n    state: Internal,\n    freq: f64,\n    value: Option<i64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let value = match value {\n        None => None,\n        Some(val) => unsafe {\n            AnyElement::from_polymorphic_datum(pg_sys::Datum::from(val), false, pg_sys::INT8OID)\n        },\n    };\n    freq_agg_trans(state, freq, value, fcinfo)\n}\n\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\npub fn freq_agg_text_trans(\n    state: Internal,\n    freq: f64,\n    value: Option<crate::raw::text>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let txt = value.map(|v| unsafe { pg_sys::pg_detoast_datum_copy(v.0.cast_mut_ptr()) });\n    let value = match txt {\n        None => None,\n        Some(val) => unsafe {\n            AnyElement::from_polymorphic_datum(pg_sys::Datum::from(val), false, pg_sys::TEXTOID)\n        },\n    };\n    freq_agg_trans(state, freq, value, fcinfo)\n}\n\npub fn space_saving_trans<F>(\n    state: Option<Inner<SpaceSavingTransState>>,\n    value: Option<AnyElement>,\n    fcinfo: pg_sys::FunctionCallInfo,\n    make_trans_state: F,\n) -> Option<Inner<SpaceSavingTransState>>\nwhere\n    F: FnOnce(pg_sys::Oid, Option<pg_sys::Oid>) -> SpaceSavingTransState,\n{\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let value = match value {\n                None => return state,\n                Some(value) => value,\n            };\n            let mut state = match state {\n                None => {\n                    let typ = value.oid();\n                    let collation = get_collation_or_default(fcinfo);\n                    make_trans_state(typ, collation).into()\n                }\n                Some(state) => state,\n            };\n\n            state.add(value.into());\n            Some(state)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn rollup_agg_trans<'input>(\n    state: Internal,\n    value: Option<SpaceSavingAggregate<'input>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let value = match value {\n        None => return Some(state),\n        Some(v) => v,\n    };\n    rollup_agg_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\n\npub fn rollup_agg_trans_inner(\n    state: Option<Inner<SpaceSavingTransState>>,\n    value: SpaceSavingAggregate,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<SpaceSavingTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let trans = (&value, &fcinfo).into();\n            if let Some(state) = state {\n                Some(SpaceSavingTransState::combine(&state, &trans).into())\n            } else {\n                Some(trans.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn rollup_agg_bigint_trans<'input>(\n    state: Internal,\n    value: Option<SpaceSavingBigIntAggregate<'input>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let value = match value {\n        None => return Some(state),\n        Some(v) => v,\n    };\n    rollup_agg_bigint_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\n\npub fn rollup_agg_bigint_trans_inner(\n    state: Option<Inner<SpaceSavingTransState>>,\n    value: SpaceSavingBigIntAggregate,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<SpaceSavingTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let trans = (&value, &fcinfo).into();\n            if let Some(state) = state {\n                Some(SpaceSavingTransState::combine(&state, &trans).into())\n            } else {\n                Some(trans.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn rollup_agg_text_trans<'input>(\n    state: Internal,\n    value: Option<SpaceSavingTextAggregate<'input>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let value = match value {\n        None => return Some(state),\n        Some(v) => v,\n    };\n    rollup_agg_text_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\n\npub fn rollup_agg_text_trans_inner(\n    state: Option<Inner<SpaceSavingTransState>>,\n    value: SpaceSavingTextAggregate,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<SpaceSavingTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let trans = (&value, &fcinfo).into();\n            if let Some(state) = state {\n                Some(SpaceSavingTransState::combine(&state, &trans).into())\n            } else {\n                Some(trans.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn space_saving_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { space_saving_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\npub fn space_saving_combine_inner(\n    a: Option<Inner<SpaceSavingTransState>>,\n    b: Option<Inner<SpaceSavingTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<SpaceSavingTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (a, b) {\n            (Some(a), Some(b)) => Some(SpaceSavingTransState::combine(&a, &b).into()),\n            (Some(a), None) => Some(a.clone().into()),\n            (None, Some(b)) => Some(b.clone().into()),\n            (None, None) => None,\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn space_saving_final(\n    state: Internal,\n    _fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<SpaceSavingAggregate<'static>> {\n    let state: Option<&SpaceSavingTransState> = unsafe { state.get() };\n    state.map(SpaceSavingAggregate::from)\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn space_saving_bigint_final(\n    state: Internal,\n    _fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<SpaceSavingBigIntAggregate<'static>> {\n    let state: Option<&SpaceSavingTransState> = unsafe { state.get() };\n    state.map(SpaceSavingBigIntAggregate::from)\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn space_saving_text_final(\n    state: Internal,\n    _fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<SpaceSavingTextAggregate<'static>> {\n    let state: Option<&SpaceSavingTransState> = unsafe { state.get() };\n    state.map(SpaceSavingTextAggregate::from)\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn space_saving_serialize(state: Internal) -> bytea {\n    let state: Inner<SpaceSavingTransState> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn space_saving_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: SpaceSavingTransState = crate::do_deserialize!(bytes, SpaceSavingTransState);\n    Inner::from(i).internal()\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.raw_freq_agg(\\n\\\n        frequency double precision, value AnyElement\\n\\\n    ) (\\n\\\n        sfunc = toolkit_experimental.freq_agg_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"freq_agg\",\n    requires = [\n        freq_agg_trans,\n        space_saving_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.freq_agg(\\n\\\n        frequency double precision, value INT8\\n\\\n    ) (\\n\\\n        sfunc = toolkit_experimental.freq_agg_bigint_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_bigint_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"freq_bigint_agg\",\n    requires = [\n        freq_agg_bigint_trans,\n        space_saving_bigint_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.freq_agg(\\n\\\n        frequency double precision, value TEXT\\n\\\n    ) (\\n\\\n        sfunc = toolkit_experimental.freq_agg_text_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_text_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"freq_text_agg\",\n    requires = [\n        freq_agg_text_trans,\n        space_saving_text_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE raw_mcv_agg(\\n\\\n        count integer, value AnyElement\\n\\\n    ) (\\n\\\n        sfunc = mcv_agg_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"mcv_agg\",\n    requires = [\n        mcv_agg_trans,\n        space_saving_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE mcv_agg(\\n\\\n        count integer, value INT8\\n\\\n    ) (\\n\\\n        sfunc = mcv_agg_bigint_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_bigint_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"mcv_bigint_agg\",\n    requires = [\n        mcv_agg_bigint_trans,\n        space_saving_bigint_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE mcv_agg(\\n\\\n        count integer, value TEXT\\n\\\n    ) (\\n\\\n        sfunc = mcv_agg_text_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_text_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"mcv_text_agg\",\n    requires = [\n        mcv_agg_text_trans,\n        space_saving_text_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE raw_mcv_agg(\\n\\\n        count integer, skew double precision, value AnyElement\\n\\\n    ) (\\n\\\n        sfunc = mcv_agg_with_skew_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"mcv_agg_with_skew\",\n    requires = [\n        mcv_agg_with_skew_trans,\n        space_saving_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE mcv_agg(\\n\\\n        count integer, skew double precision, value int8\\n\\\n    ) (\\n\\\n        sfunc = mcv_agg_with_skew_bigint_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_bigint_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"mcv_agg_with_skew_bigint\",\n    requires = [\n        mcv_agg_with_skew_bigint_trans,\n        space_saving_bigint_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE mcv_agg(\\n\\\n        count integer, skew double precision, value text\\n\\\n    ) (\\n\\\n        sfunc = mcv_agg_with_skew_text_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_text_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"mcv_agg_with_skew_text\",\n    requires = [\n        mcv_agg_with_skew_text_trans,\n        space_saving_text_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        agg SpaceSavingAggregate\\n\\\n    ) (\\n\\\n        sfunc = rollup_agg_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"freq_agg_rollup\",\n    requires = [\n        rollup_agg_trans,\n        space_saving_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        agg SpaceSavingBigIntAggregate\\n\\\n    ) (\\n\\\n        sfunc = rollup_agg_bigint_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_bigint_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"freq_agg_bigint_rollup\",\n    requires = [\n        rollup_agg_bigint_trans,\n        space_saving_bigint_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        agg SpaceSavingTextAggregate\\n\\\n    ) (\\n\\\n        sfunc = rollup_agg_text_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = space_saving_text_final,\\n\\\n        combinefunc = space_saving_combine,\\n\\\n        serialfunc = space_saving_serialize,\\n\\\n        deserialfunc = space_saving_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"freq_agg_text_rollup\",\n    requires = [\n        rollup_agg_text_trans,\n        space_saving_text_final,\n        space_saving_combine,\n        space_saving_serialize,\n        space_saving_deserialize\n    ],\n);\n\n#[pg_extern(immutable, parallel_safe, name = \"into_values\")]\npub fn freq_iter<'a>(\n    agg: SpaceSavingAggregate<'a>,\n    ty: AnyElement,\n) -> TableIterator<\n    'a,\n    (\n        name!(value, AnyElement),\n        name!(min_freq, f64),\n        name!(max_freq, f64),\n    ),\n> {\n    unsafe {\n        if ty.oid().to_u32() != agg.type_oid {\n            pgrx::error!(\"mischatched types\")\n        }\n        let counts = agg.counts.slice().iter().zip(agg.overcounts.slice().iter());\n        TableIterator::new(agg.datums.clone().into_iter().zip(counts).map_while(\n            move |(value, (&count, &overcount))| {\n                let total = agg.values_seen as f64;\n                let value =\n                    AnyElement::from_polymorphic_datum(value, false, Oid::from(agg.type_oid))\n                        .unwrap();\n                let min_freq = (count - overcount) as f64 / total;\n                let max_freq = count as f64 / total;\n                Some((value, min_freq, max_freq))\n            },\n        ))\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"into_values\")]\npub fn freq_bigint_iter<'a>(\n    agg: SpaceSavingBigIntAggregate<'a>,\n) -> TableIterator<\n    'a,\n    (\n        name!(value, i64),\n        name!(min_freq, f64),\n        name!(max_freq, f64),\n    ),\n> {\n    let counts = agg.counts.slice().iter().zip(agg.overcounts.slice().iter());\n    TableIterator::new(agg.datums.clone().into_iter().zip(counts).map_while(\n        move |(value, (&count, &overcount))| {\n            let total = agg.values_seen as f64;\n            let min_freq = (count - overcount) as f64 / total;\n            let max_freq = count as f64 / total;\n            Some((value, min_freq, max_freq))\n        },\n    ))\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_freq_bigint_iter<'a>(\n    agg: SpaceSavingBigIntAggregate<'a>,\n    _accessor: AccessorIntoValues,\n) -> TableIterator<\n    'a,\n    (\n        name!(value, i64),\n        name!(min_freq, f64),\n        name!(max_freq, f64),\n    ),\n> {\n    freq_bigint_iter(agg)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"into_values\")]\npub fn freq_text_iter<'a>(\n    agg: SpaceSavingTextAggregate<'a>,\n) -> TableIterator<\n    'a,\n    (\n        name!(value, String),\n        name!(min_freq, f64),\n        name!(max_freq, f64),\n    ),\n> {\n    let counts = agg.counts.slice().iter().zip(agg.overcounts.slice().iter());\n    TableIterator::new(agg.datums.clone().into_iter().zip(counts).map_while(\n        move |(value, (&count, &overcount))| {\n            let total = agg.values_seen as f64;\n            let data = unsafe { varlena_to_string(value.cast_mut_ptr()) };\n            let min_freq = (count - overcount) as f64 / total;\n            let max_freq = count as f64 / total;\n            Some((data, min_freq, max_freq))\n        },\n    ))\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_freq_text_iter<'a>(\n    agg: SpaceSavingTextAggregate<'a>,\n    _accessor: AccessorIntoValues,\n) -> TableIterator<\n    'a,\n    (\n        name!(value, String),\n        name!(min_freq, f64),\n        name!(max_freq, f64),\n    ),\n> {\n    freq_text_iter(agg)\n}\n\nfn validate_topn_for_mcv_agg(\n    n: i32,\n    topn: u32,\n    skew: f64,\n    total_vals: u64,\n    counts: impl Iterator<Item = u64>,\n) {\n    if topn == 0 {\n        // Not a mcv aggregate\n        return;\n    }\n\n    // TODO: should we allow this if we have enough data?\n    if n > topn as i32 {\n        pgrx::error!(\n            \"requested N ({}) exceeds creation parameter of mcv aggregate ({})\",\n            n,\n            topn\n        )\n    }\n\n    // For mcv_aggregates distributions we check that the top 'n' values satisfy the cumulative distribution\n    // for our zeta curve.\n    let needed_count = (zeta_le_n(skew, n as u64) * total_vals as f64).ceil() as u64;\n    if counts.take(n as usize).sum::<u64>() < needed_count {\n        pgrx::error!(\"data is not skewed enough to find top {} parameters with a skew of {}, try reducing the skew factor\", n , skew)\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn topn(\n    agg: SpaceSavingAggregate<'_>,\n    n: i32,\n    ty: Option<AnyElement>,\n) -> SetOfIterator<'_, AnyElement> {\n    // If called with a NULL, assume type matches\n    if ty.is_some() && ty.unwrap().oid().to_u32() != agg.type_oid {\n        pgrx::error!(\"mischatched types\")\n    }\n\n    validate_topn_for_mcv_agg(\n        n,\n        agg.topn as u32,\n        agg.freq_param,\n        agg.values_seen,\n        agg.counts.iter(),\n    );\n    let min_freq = if agg.topn == 0 { agg.freq_param } else { 0. };\n\n    let type_oid: u32 = agg.type_oid;\n    SetOfIterator::new(\n        TopNIterator::new(\n            agg.datums.clone().into_iter(),\n            agg.counts.clone().into_vec(),\n            agg.values_seen as f64,\n            n,\n            min_freq,\n        )\n        // TODO Shouldn't failure to convert to AnyElement cause error, not early stop?\n        .map_while(move |value| unsafe {\n            AnyElement::from_polymorphic_datum(value, false, Oid::from(type_oid))\n        }),\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"topn\")]\npub fn default_topn(\n    agg: SpaceSavingAggregate<'_>,\n    ty: Option<AnyElement>,\n) -> SetOfIterator<'_, AnyElement> {\n    if agg.topn == 0 {\n        pgrx::error!(\"frequency aggregates require a N parameter to topn\")\n    }\n    let n = agg.topn as i32;\n    topn(agg, n, ty)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"topn\")]\npub fn topn_bigint(agg: SpaceSavingBigIntAggregate<'_>, n: i32) -> SetOfIterator<'_, i64> {\n    validate_topn_for_mcv_agg(\n        n,\n        agg.topn,\n        agg.freq_param,\n        agg.values_seen,\n        agg.counts.iter(),\n    );\n    let min_freq = if agg.topn == 0 { agg.freq_param } else { 0. };\n\n    SetOfIterator::new(TopNIterator::new(\n        agg.datums.clone().into_iter(),\n        agg.counts.clone().into_vec(),\n        agg.values_seen as f64,\n        n,\n        min_freq,\n    ))\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_topn_bigint<'a>(\n    agg: SpaceSavingBigIntAggregate<'a>,\n    accessor: AccessorTopNCount,\n) -> SetOfIterator<'a, i64> {\n    topn_bigint(agg, accessor.count as i32)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"topn\")]\npub fn default_topn_bigint(agg: SpaceSavingBigIntAggregate<'_>) -> SetOfIterator<'_, i64> {\n    if agg.topn == 0 {\n        pgrx::error!(\"frequency aggregates require a N parameter to topn\")\n    }\n    let n = agg.topn as i32;\n    topn_bigint(agg, n)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_default_topn_bigint<'a>(\n    agg: SpaceSavingBigIntAggregate<'a>,\n    _accessor: AccessorTopn,\n) -> SetOfIterator<'a, i64> {\n    default_topn_bigint(agg)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"topn\")]\npub fn topn_text(agg: SpaceSavingTextAggregate<'_>, n: i32) -> SetOfIterator<'_, String> {\n    validate_topn_for_mcv_agg(\n        n,\n        agg.topn,\n        agg.freq_param,\n        agg.values_seen,\n        agg.counts.iter(),\n    );\n    let min_freq = if agg.topn == 0 { agg.freq_param } else { 0. };\n\n    SetOfIterator::new(\n        TopNIterator::new(\n            agg.datums.clone().into_iter(),\n            agg.counts.clone().into_vec(),\n            agg.values_seen as f64,\n            n,\n            min_freq,\n        )\n        .map(|value| unsafe { varlena_to_string(value.cast_mut_ptr()) }),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_topn_text<'a>(\n    agg: SpaceSavingTextAggregate<'a>,\n    accessor: AccessorTopNCount,\n) -> SetOfIterator<'a, String> {\n    topn_text(agg, accessor.count as i32)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"topn\")]\npub fn default_topn_text(agg: SpaceSavingTextAggregate<'_>) -> SetOfIterator<'_, String> {\n    if agg.topn == 0 {\n        pgrx::error!(\"frequency aggregates require a N parameter to topn\")\n    }\n    let n = agg.topn as i32;\n    topn_text(agg, n)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_default_topn_text<'a>(\n    agg: SpaceSavingTextAggregate<'a>,\n    _accessor: AccessorTopn,\n) -> SetOfIterator<'a, String> {\n    default_topn_text(agg)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_frequency(agg: SpaceSavingAggregate<'_>, value: AnyElement) -> f64 {\n    let value: PgAnyElement = value.into();\n    match agg\n        .datums\n        .iter()\n        .position(|datum| value == (datum, Oid::from(agg.type_oid)).into())\n    {\n        Some(idx) => agg.counts.slice()[idx] as f64 / agg.values_seen as f64,\n        None => 0.,\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_frequency(agg: SpaceSavingAggregate<'_>, value: AnyElement) -> f64 {\n    let value: PgAnyElement = value.into();\n    match agg\n        .datums\n        .iter()\n        .position(|datum| value == (datum, Oid::from(agg.type_oid)).into())\n    {\n        Some(idx) => {\n            (agg.counts.slice()[idx] - agg.overcounts.slice()[idx]) as f64 / agg.values_seen as f64\n        }\n        None => 0.,\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"max_frequency\")]\npub fn max_bigint_frequency(agg: SpaceSavingBigIntAggregate<'_>, value: i64) -> f64 {\n    match agg.datums.iter().position(|datum| value == datum) {\n        Some(idx) => agg.counts.slice()[idx] as f64 / agg.values_seen as f64,\n        None => 0.,\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_bigint_frequency<'a>(\n    agg: SpaceSavingBigIntAggregate<'a>,\n    accessor: AccessorMaxFrequencyInt,\n) -> f64 {\n    max_bigint_frequency(agg, accessor.value)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"min_frequency\")]\npub fn min_bigint_frequency(agg: SpaceSavingBigIntAggregate<'_>, value: i64) -> f64 {\n    match agg.datums.iter().position(|datum| value == datum) {\n        Some(idx) => {\n            (agg.counts.slice()[idx] - agg.overcounts.slice()[idx]) as f64 / agg.values_seen as f64\n        }\n        None => 0.,\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_bigint_frequency<'a>(\n    agg: SpaceSavingBigIntAggregate<'a>,\n    accessor: AccessorMinFrequencyInt,\n) -> f64 {\n    min_bigint_frequency(agg, accessor.value)\n}\n\n// Still needs an arrow operator defined, but the text datum input is a bit finicky.\n#[pg_extern(immutable, parallel_safe, name = \"max_frequency\")]\npub fn max_text_frequency(agg: SpaceSavingTextAggregate<'_>, value: text) -> f64 {\n    let value: PgAnyElement = (value.0, pg_sys::TEXTOID).into();\n    match agg\n        .datums\n        .iter()\n        .position(|datum| value == (datum, pg_sys::TEXTOID).into())\n    {\n        Some(idx) => agg.counts.slice()[idx] as f64 / agg.values_seen as f64,\n        None => 0.,\n    }\n}\n\n// Still needs an arrow operator defined, but the text datum input is a bit finicky.\n#[pg_extern(immutable, parallel_safe, name = \"min_frequency\")]\npub fn min_text_frequency(agg: SpaceSavingTextAggregate<'_>, value: text) -> f64 {\n    let value: PgAnyElement = (value.0, pg_sys::TEXTOID).into();\n    match agg\n        .datums\n        .iter()\n        .position(|datum| value == (datum, pg_sys::TEXTOID).into())\n    {\n        Some(idx) => {\n            (agg.counts.slice()[idx] - agg.overcounts.slice()[idx]) as f64 / agg.values_seen as f64\n        }\n        None => 0.,\n    }\n}\n\nstruct TopNIterator<Input, InputIterator: std::iter::Iterator<Item = Input>> {\n    datums_iter: InputIterator,\n    counts_iter: std::vec::IntoIter<u64>,\n    values_seen: f64,\n    max_n: u32,\n    min_freq: f64,\n    i: u32,\n}\n\nimpl<Input, InputIterator: std::iter::Iterator<Item = Input>> TopNIterator<Input, InputIterator> {\n    fn new(\n        datums_iter: InputIterator,\n        counts: Vec<u64>,\n        values_seen: f64,\n        max_n: i32,\n        min_freq: f64,\n    ) -> Self {\n        Self {\n            datums_iter,\n            counts_iter: counts.into_iter(),\n            values_seen,\n            max_n: max_n as u32,\n            min_freq,\n            i: 0,\n        }\n    }\n}\n\nimpl<Input, InputIterator: std::iter::Iterator<Item = Input>> Iterator\n    for TopNIterator<Input, InputIterator>\n{\n    type Item = Input;\n    fn next(&mut self) -> Option<Self::Item> {\n        match (self.datums_iter.next(), self.counts_iter.next()) {\n            (Some(value), Some(count)) => {\n                self.i += 1;\n                if self.i > self.max_n || count as f64 / self.values_seen < self.min_freq {\n                    None\n                } else {\n                    Some(value)\n                }\n            }\n            _ => None,\n        }\n    }\n}\n\nunsafe fn varlena_to_string(vl: *const pg_sys::varlena) -> String {\n    let bytes: &[u8] = varlena_to_byte_slice(vl);\n    let s = std::str::from_utf8(bytes).expect(\"Error creating string from text data\");\n    s.into()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n    use rand::distributions::{Distribution, Uniform};\n    use rand::prelude::SliceRandom;\n    use rand::thread_rng;\n    use rand::RngCore;\n    use rand_distr::Zeta;\n\n    #[pg_test]\n    fn test_freq_aggregate() {\n        Spi::connect_mut(|client| {\n            // using the search path trick for this test to make it easier to stabilize later on\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE test (data INTEGER, time TIMESTAMPTZ)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in (0..100).rev() {\n                client.update(&format!(\"INSERT INTO test SELECT i, '2020-1-1'::TIMESTAMPTZ + ('{} days, ' || i::TEXT || ' seconds')::INTERVAL FROM generate_series({i}, 99, 1) i\", 100 - i), None, &[]).unwrap();\n            }\n\n            let test = client.update(\"SELECT freq_agg(0.015, s.data)::TEXT FROM (SELECT data FROM test ORDER BY time) s\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap().unwrap();\n            let expected = \"(version:1,num_values:67,topn:0,values_seen:5050,freq_param:0.015,counts:[100,99,98,97,96,95,94,93,92,91,90,89,88,87,86,85,84,83,82,81,80,79,78,77,76,75,74,73,72,71,70,69,68,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67],overcounts:[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66],datums:[99,98,97,96,95,94,93,92,91,90,89,88,87,86,85,84,83,82,81,80,79,78,77,76,75,74,73,72,71,70,69,68,67,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66])\";\n            assert_eq!(test, expected);\n\n            let test = client.update(\"SELECT raw_freq_agg(0.015, s.data)::TEXT FROM (SELECT data FROM test ORDER BY time) s\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap().unwrap();\n            let expected = \"(version:1,type_oid:23,num_values:67,values_seen:5050,freq_param:0.015,topn:0,counts:[100,99,98,97,96,95,94,93,92,91,90,89,88,87,86,85,84,83,82,81,80,79,78,77,76,75,74,73,72,71,70,69,68,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67,67],overcounts:[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66,66],datums:[23,\\\"99\\\",\\\"98\\\",\\\"97\\\",\\\"96\\\",\\\"95\\\",\\\"94\\\",\\\"93\\\",\\\"92\\\",\\\"91\\\",\\\"90\\\",\\\"89\\\",\\\"88\\\",\\\"87\\\",\\\"86\\\",\\\"85\\\",\\\"84\\\",\\\"83\\\",\\\"82\\\",\\\"81\\\",\\\"80\\\",\\\"79\\\",\\\"78\\\",\\\"77\\\",\\\"76\\\",\\\"75\\\",\\\"74\\\",\\\"73\\\",\\\"72\\\",\\\"71\\\",\\\"70\\\",\\\"69\\\",\\\"68\\\",\\\"67\\\",\\\"33\\\",\\\"34\\\",\\\"35\\\",\\\"36\\\",\\\"37\\\",\\\"38\\\",\\\"39\\\",\\\"40\\\",\\\"41\\\",\\\"42\\\",\\\"43\\\",\\\"44\\\",\\\"45\\\",\\\"46\\\",\\\"47\\\",\\\"48\\\",\\\"49\\\",\\\"50\\\",\\\"51\\\",\\\"52\\\",\\\"53\\\",\\\"54\\\",\\\"55\\\",\\\"56\\\",\\\"57\\\",\\\"58\\\",\\\"59\\\",\\\"60\\\",\\\"61\\\",\\\"62\\\",\\\"63\\\",\\\"64\\\",\\\"65\\\",\\\"66\\\"])\";\n            assert_eq!(test, expected);\n        });\n    }\n\n    #[pg_test]\n    fn test_topn_aggregate() {\n        Spi::connect_mut(|client| {\n            // using the search path trick for this test to make it easier to stabilize later on\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE test (data INTEGER, time TIMESTAMPTZ)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in (0..200).rev() {\n                client.update(&format!(\"INSERT INTO test SELECT i, '2020-1-1'::TIMESTAMPTZ + ('{} days, ' || i::TEXT || ' seconds')::INTERVAL FROM generate_series({i}, 199, 1) i\", 200 - i), None, &[]).unwrap();\n            }\n\n            let test = client\n                .update(\n                    \"SELECT mcv_agg(10, s.data)::TEXT FROM (SELECT data FROM test ORDER BY time) s\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            let expected = \"(version:1,num_values:110,topn:10,values_seen:20100,freq_param:1.1,counts:[200,199,198,197,196,195,194,193,192,191,190,189,188,187,186,185,184,183,182,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181,181],overcounts:[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180,180],datums:[199,198,197,196,195,194,193,192,191,190,189,188,187,186,185,184,183,182,181,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180])\";\n            assert_eq!(test, expected);\n        });\n    }\n\n    #[pg_test]\n    fn explicit_aggregate_test() {\n        let freq = 0.0625;\n        let fcinfo = std::ptr::null_mut(); // dummy value, will use default collation\n        let mut state = None.into();\n\n        for i in 11..=20 {\n            for j in i..=20 {\n                let value = unsafe {\n                    AnyElement::from_polymorphic_datum(\n                        pg_sys::Datum::from(j),\n                        false,\n                        pg_sys::INT4OID,\n                    )\n                };\n                state = super::freq_agg_trans(state, freq, value, fcinfo).unwrap();\n            }\n        }\n\n        let first = super::space_saving_serialize(state);\n\n        let bytes = unsafe {\n            std::slice::from_raw_parts(\n                vardata_any(first.0.cast_mut_ptr()) as *const u8,\n                varsize_any_exhdr(first.0.cast_mut_ptr()),\n            )\n        };\n        let expected = [\n            1, 1, // versions\n            15, 0, 0, 0, 0, 0, 0, 0, // size hint for sequence\n            55, 0, 0, 0, 0, 0, 0, 0, // elements seen\n            0, 0, 0, 0, 0, 0, 176, 63, // frequency (f64 encoding of 0.0625)\n            17, 0, 0, 0, // elements tracked\n            0, 0, 0, 0, // topn\n            7, 0, 0, 0, 1, 1, 10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111,\n            103, 11, 0, 0, 0, 0, 0, 0, 0, 101, 110, 95, 85, 83, 46, 85, 84, 70, 45,\n            56, // INT4 hasher\n            2, 0, 0, 0, 0, 0, 0, 0, 50, 48, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 20, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 57, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 19, count 9, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 56, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 18, count 8, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 55, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 17, count 7, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 54, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 16, count 6, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 53, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 15, count 5, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 52, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 14, count 4, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 51, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 13, count 3, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 50, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 12, count 2, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 49, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 11, count 1, overcount 0\n        ];\n        // encoding of hasher can vary on platform and across postgres version (even in length), ignore it and check the other fields\n        let prefix_len = 8 * 4 + 2;\n        let suffix_len = (8 + 2 + 16) * 10;\n        assert_eq!(bytes[..prefix_len], expected[..prefix_len]);\n        assert_eq!(\n            bytes[bytes.len() - suffix_len..],\n            expected[expected.len() - suffix_len..]\n        );\n\n        state = None.into();\n\n        for i in (1..=10).rev() {\n            // reverse here introduces less error in the aggregate\n            for j in i..=20 {\n                let value = unsafe {\n                    AnyElement::from_polymorphic_datum(\n                        pg_sys::Datum::from(j),\n                        false,\n                        pg_sys::INT4OID,\n                    )\n                };\n                state = super::freq_agg_trans(state, freq, value, fcinfo).unwrap();\n            }\n        }\n\n        let second = super::space_saving_serialize(state);\n\n        let bytes: &[u8] = unsafe {\n            std::slice::from_raw_parts(\n                vardata_any(second.0.cast_mut_ptr()) as *const u8,\n                varsize_any_exhdr(second.0.cast_mut_ptr()),\n            )\n        };\n        let expected: [u8; 513] = [\n            1, 1, // versions\n            22, 0, 0, 0, 0, 0, 0, 0, // size hint for sequence\n            155, 0, 0, 0, 0, 0, 0, 0, // elements seen\n            0, 0, 0, 0, 0, 0, 176, 63, // frequency (f64 encoding of 0.0625)\n            17, 0, 0, 0, // elements tracked\n            0, 0, 0, 0, // topn\n            7, 0, 0, 0, 1, 1, 10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111,\n            103, 11, 0, 0, 0, 0, 0, 0, 0, 101, 110, 95, 85, 83, 46, 85, 84, 70, 45,\n            56, // INT4 hasher\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 48, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 10, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 49, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 11, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 50, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 12, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 51, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 13, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 52, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 14, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 53, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 15, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 54, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 16, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 55, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 17, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 56, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 18, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 57, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 19, count 10, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 50, 48, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 20, count 10, overcount 0\n            1, 0, 0, 0, 0, 0, 0, 0, 57, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 9, count 9, overcount 0\n            1, 0, 0, 0, 0, 0, 0, 0, 56, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 8, count 8, overcount 0\n            1, 0, 0, 0, 0, 0, 0, 0, 52, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 4, count 7, overcount 6\n            1, 0, 0, 0, 0, 0, 0, 0, 53, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 5, count 7, overcount 6\n            1, 0, 0, 0, 0, 0, 0, 0, 54, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 6, count 7, overcount 6\n            1, 0, 0, 0, 0, 0, 0, 0, 55, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 7, count 7, overcount 6\n        ];\n        // encoding of hasher can vary on platform and across postgres version (even in length), ignore it and check the other fields\n        let suffix_len = (8 + 2 + 16) * 11 + (8 + 1 + 16) * 6;\n        assert_eq!(bytes[..prefix_len], expected[..prefix_len]);\n        assert_eq!(\n            bytes[bytes.len() - suffix_len..],\n            expected[expected.len() - suffix_len..]\n        );\n\n        let combined = super::space_saving_serialize(\n            super::space_saving_combine(\n                super::space_saving_deserialize(first, None.into()).unwrap(),\n                super::space_saving_deserialize(second, None.into()).unwrap(),\n                fcinfo,\n            )\n            .unwrap(),\n        );\n\n        let bytes = unsafe {\n            std::slice::from_raw_parts(\n                vardata_any(combined.0.cast_mut_ptr()) as *const u8,\n                varsize_any_exhdr(combined.0.cast_mut_ptr()),\n            )\n        };\n        let expected: [u8; 513] = [\n            1, 1, // versions\n            22, 0, 0, 0, 0, 0, 0, 0, // size hint for sequence\n            210, 0, 0, 0, 0, 0, 0, 0, // elements seen\n            0, 0, 0, 0, 0, 0, 176, 63, // frequency (f64 encoding of 0.0625)\n            17, 0, 0, 0, // elements tracked\n            0, 0, 0, 0, // topn\n            7, 0, 0, 0, 1, 1, 10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111,\n            103, 11, 0, 0, 0, 0, 0, 0, 0, 101, 110, 95, 85, 83, 46, 85, 84, 70, 45,\n            56, // INT4 hasher\n            2, 0, 0, 0, 0, 0, 0, 0, 50, 48, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 20, count 20, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 57, 19, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 19, count 19, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 56, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 18, count 18, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 55, 17, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 17, count 17, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 54, 16, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 16, count 16, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 53, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 15, count 15, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 52, 14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 14, count 14, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 51, 13, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 13, count 13, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 50, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 12, count 12, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 49, 11, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 11, count 11, overcount 0\n            2, 0, 0, 0, 0, 0, 0, 0, 49, 48, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 10, count 10, overcount 0\n            1, 0, 0, 0, 0, 0, 0, 0, 57, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 9, count 9, overcount 0\n            1, 0, 0, 0, 0, 0, 0, 0, 56, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n            0, // string 8, count 8, overcount 0\n            1, 0, 0, 0, 0, 0, 0, 0, 52, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 4, count 7, overcount 6\n            1, 0, 0, 0, 0, 0, 0, 0, 54, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 6, count 7, overcount 6\n            1, 0, 0, 0, 0, 0, 0, 0, 53, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 5, count 7, overcount 6\n            1, 0, 0, 0, 0, 0, 0, 0, 55, 7, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0,\n            0, // string 7, count 7, overcount 6\n        ];\n        // encoding of hasher can vary on platform and across postgres version (even in length), ignore it and check the other fields\n        let suffix_len = (8 + 2 + 16) * 11 + (8 + 1 + 16) * 6;\n        assert_eq!(bytes[..prefix_len], expected[..prefix_len]);\n        assert_eq!(\n            bytes[bytes.len() - suffix_len..],\n            expected[expected.len() - suffix_len..]\n        );\n    }\n\n    // Setup environment and create table 'test' with some aggregates in table 'aggs'\n    fn setup_with_test_table(client: &mut pgrx::spi::SpiClient) {\n        // using the search path trick for this test to make it easier to stabilize later on\n        let sp = client\n            .update(\n                \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                None,\n                &[],\n            )\n            .unwrap()\n            .first()\n            .get_one::<String>()\n            .unwrap()\n            .unwrap();\n        client\n            .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n            .unwrap();\n\n        client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n        client\n            .update(\n                \"CREATE TABLE test (data INTEGER, time TIMESTAMPTZ)\",\n                None,\n                &[],\n            )\n            .unwrap();\n\n        for i in (0..20).rev() {\n            client.update(&format!(\"INSERT INTO test SELECT i, '2020-1-1'::TIMESTAMPTZ + ('{} days, ' || i::TEXT || ' seconds')::INTERVAL FROM generate_series({i}, 19, 1) i\", 10 - i), None, &[]).unwrap();\n        }\n\n        client\n            .update(\n                \"CREATE TABLE aggs (name TEXT, agg SPACESAVINGBIGINTAGGREGATE)\",\n                None,\n                &[],\n            )\n            .unwrap();\n        client.update(\"INSERT INTO aggs SELECT 'mcv_default', mcv_agg(5, s.data) FROM (SELECT data FROM test ORDER BY time) s\", None, &[]).unwrap();\n        client.update(\"INSERT INTO aggs SELECT 'mcv_1.5', mcv_agg(5, 1.5, s.data) FROM (SELECT data FROM test ORDER BY time) s\", None, &[]).unwrap();\n        client.update(\"INSERT INTO aggs SELECT 'mcv_2', mcv_agg(5, 2, s.data) FROM (SELECT data FROM test ORDER BY time) s\", None, &[]).unwrap();\n        client.update(\"INSERT INTO aggs SELECT 'freq_8', freq_agg(0.08, s.data) FROM (SELECT data FROM test ORDER BY time) s\", None, &[]).unwrap();\n        client.update(\"INSERT INTO aggs SELECT 'freq_5', freq_agg(0.05, s.data) FROM (SELECT data FROM test ORDER BY time) s\", None, &[]).unwrap();\n        client.update(\"INSERT INTO aggs SELECT 'freq_2', freq_agg(0.02, s.data) FROM (SELECT data FROM test ORDER BY time) s\", None, &[]).unwrap();\n    }\n\n    // API tests\n    #[pg_test]\n    fn test_topn() {\n        Spi::connect_mut(|client| {\n            setup_with_test_table(client);\n\n            // simple tests\n            let rows = client\n                .update(\n                    \"SELECT topn(agg) FROM aggs WHERE name = 'mcv_default'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 5);\n            let rows = client\n                .update(\n                    \"SELECT agg -> topn() FROM aggs WHERE name = 'mcv_default'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 5);\n            let rows = client\n                .update(\n                    \"SELECT topn(agg, 5) FROM aggs WHERE name = 'freq_5'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 5);\n\n            // can limit below topn_agg value\n            let rows = client\n                .update(\n                    \"SELECT topn(agg, 3) FROM aggs WHERE name = 'mcv_default'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 3);\n            let rows = client\n                .update(\n                    \"SELECT agg -> topn(3) FROM aggs WHERE name = 'mcv_default'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 3);\n\n            // only 4 rows with freq >= 0.08\n            let rows = client\n                .update(\n                    \"SELECT topn(agg, 5) FROM aggs WHERE name = 'freq_8'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 4);\n        });\n    }\n\n    #[pg_test(\n        error = \"data is not skewed enough to find top 0 parameters with a skew of 1.5, try reducing the skew factor\"\n    )]\n    fn topn_on_underskewed_mcv_agg() {\n        Spi::connect_mut(|client| {\n            setup_with_test_table(client);\n            client\n                .update(\n                    \"SELECT topn(agg, 0::int) FROM aggs WHERE name = 'mcv_1.5'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n        });\n    }\n\n    #[pg_test(error = \"requested N (8) exceeds creation parameter of mcv aggregate (5)\")]\n    fn topn_high_n_on_mcv_agg() {\n        Spi::connect_mut(|client| {\n            setup_with_test_table(client);\n            client\n                .update(\n                    \"SELECT topn(agg, 8) FROM aggs WHERE name = 'mcv_default'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n        });\n    }\n\n    #[pg_test(error = \"frequency aggregates require a N parameter to topn\")]\n    fn topn_requires_n_for_freq_agg() {\n        Spi::connect_mut(|client| {\n            setup_with_test_table(client);\n            assert_eq!(\n                0,\n                client\n                    .update(\n                        \"SELECT topn(agg) FROM aggs WHERE name = 'freq_2'\",\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .count(),\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_into_values() {\n        Spi::connect_mut(|client| {\n            setup_with_test_table(client);\n\n            let rows = client\n                .update(\n                    \"SELECT into_values(agg) FROM aggs WHERE name = 'freq_8'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 13);\n            let rows = client\n                .update(\n                    \"SELECT into_values(agg) FROM aggs WHERE name = 'freq_5'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 20);\n            let rows = client\n                .update(\n                    \"SELECT into_values(agg) FROM aggs WHERE name = 'freq_2'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 20);\n\n            let rows = client\n                .update(\n                    \"SELECT agg -> into_values() FROM aggs WHERE name = 'freq_8'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 13);\n            let rows = client\n                .update(\n                    \"SELECT agg -> into_values() FROM aggs WHERE name = 'freq_5'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 20);\n            let rows = client\n                .update(\n                    \"SELECT agg -> into_values() FROM aggs WHERE name = 'freq_2'\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .count();\n            assert_eq!(rows, 20);\n        });\n    }\n\n    #[pg_test]\n    fn test_frequency_getters() {\n        Spi::connect_mut(|client| {\n            setup_with_test_table(client);\n\n            // simple tests\n            let (min, max) = client.update(\"SELECT min_frequency(agg, 3), max_frequency(agg, 3) FROM aggs WHERE name = 'freq_2'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.01904761904761905);\n            assert_eq!(max.unwrap(), 0.01904761904761905);\n\n            let (min, max) = client.update(\"SELECT min_frequency(agg, 11), max_frequency(agg, 11) FROM aggs WHERE name = 'mcv_default'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.05714285714285714);\n            assert_eq!(max.unwrap(), 0.05714285714285714);\n            let (min, max) = client.update(\"SELECT agg -> min_frequency(3), agg -> max_frequency(3) FROM aggs WHERE name = 'freq_2'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.01904761904761905);\n            assert_eq!(max.unwrap(), 0.01904761904761905);\n\n            let (min, max) = client.update(\"SELECT agg -> min_frequency(11), agg -> max_frequency(11) FROM aggs WHERE name = 'mcv_default'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.05714285714285714);\n            assert_eq!(max.unwrap(), 0.05714285714285714);\n\n            // missing value\n            let (min, max) = client.update(\"SELECT min_frequency(agg, 3), max_frequency(agg, 3) FROM aggs WHERE name = 'freq_8'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.);\n            assert_eq!(max.unwrap(), 0.);\n\n            let (min, max) = client.update(\"SELECT min_frequency(agg, 20), max_frequency(agg, 20) FROM aggs WHERE name = 'mcv_2'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.);\n            assert_eq!(max.unwrap(), 0.);\n\n            // noisy value\n            let (min, max) = client.update(\"SELECT min_frequency(agg, 8), max_frequency(agg, 8) FROM aggs WHERE name = 'mcv_1.5'\", None, &[])\n                .unwrap().first()\n                .get_two::<f64,f64>().unwrap();\n            assert_eq!(min.unwrap(), 0.004761904761904762);\n            assert_eq!(max.unwrap(), 0.05238095238095238);\n        });\n    }\n\n    #[pg_test]\n    fn test_rollups() {\n        Spi::connect_mut(|client| {\n            client.update(\n                \"CREATE TABLE test (raw_data DOUBLE PRECISION, int_data INTEGER, text_data TEXT, bucket INTEGER)\",\n                None,\n                &[]\n            ).unwrap();\n\n            // Generate an array of 10000 values by taking the probability curve for a\n            // zeta curve with an s of 1.1 for the top 5 values, then adding smaller\n            // amounts of the next 5 most common values, and finally filling with unique values.\n            let mut vals = vec![1; 945];\n            vals.append(&mut vec![2; 441]);\n            vals.append(&mut vec![3; 283]);\n            vals.append(&mut vec![4; 206]);\n            vals.append(&mut vec![5; 161]);\n            for v in 6..=10 {\n                vals.append(&mut vec![v, 125]);\n            }\n            for v in 0..(10000 - 945 - 441 - 283 - 206 - 161 - (5 * 125)) {\n                vals.push(11 + v);\n            }\n            vals.shuffle(&mut thread_rng());\n\n            // Probably not the most efficient way of populating this table...\n            for v in vals {\n                let cmd = format!(\n                    \"INSERT INTO test SELECT {v}, {v}::INT, {v}::TEXT, FLOOR(RANDOM() * 10)\"\n                );\n                client.update(&cmd, None, &[]).unwrap();\n            }\n\n            // No matter how the values are batched into subaggregates, we should always\n            // see the same top 5 values\n            let mut result = client.update(\n                \"WITH aggs AS (SELECT bucket, raw_mcv_agg(5, raw_data) as raw_agg FROM test GROUP BY bucket)\n                SELECT topn(rollup(raw_agg), NULL::DOUBLE PRECISION)::TEXT from aggs\",\n                None, &[]\n            ).unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"1\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"2\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"3\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"4\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"5\"));\n            assert!(result.next().is_none());\n\n            let mut result = client.update(\n                \"WITH aggs AS (SELECT bucket, mcv_agg(5, int_data) as int_agg FROM test GROUP BY bucket)\n                SELECT topn(rollup(int_agg))::TEXT from aggs\",\n                None, &[]\n            ).unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"1\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"2\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"3\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"4\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"5\"));\n            assert!(result.next().is_none());\n\n            let mut result = client.update(\n                \"WITH aggs AS (SELECT bucket, mcv_agg(5, text_data) as text_agg FROM test GROUP BY bucket)\n                SELECT topn(rollup(text_agg))::TEXT from aggs\",\n                None, &[]\n            ).unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"1\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"2\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"3\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"4\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"5\"));\n            assert!(result.next().is_none());\n        });\n    }\n\n    #[pg_test]\n    fn test_freq_agg_invariant() {\n        // The frequency agg invariant is that any element with frequency >= f will appear in the freq_agg(f)\n\n        // This test will randomly generate 200 values in the uniform range [0, 99] and check to see any value\n        // that shows up at least 3 times appears in a frequency aggregate created with freq = 0.015\n        let rand100 = Uniform::new_inclusive(0, 99);\n        let mut rng = rand::thread_rng();\n\n        let mut counts = [0; 100];\n\n        let mut state = None.into();\n        let freq = 0.015;\n        let fcinfo = std::ptr::null_mut(); // dummy value, will use default collation\n\n        for _ in 0..200 {\n            let v = rand100.sample(&mut rng);\n            let value = unsafe {\n                AnyElement::from_polymorphic_datum(pg_sys::Datum::from(v), false, pg_sys::INT4OID)\n            };\n            state = super::freq_agg_trans(state, freq, value, fcinfo).unwrap();\n            counts[v] += 1;\n        }\n\n        let state = space_saving_final(state, fcinfo).unwrap();\n        let vals: std::collections::HashSet<usize> =\n            state.datums.iter().map(|datum| datum.value()).collect();\n\n        for (val, &count) in counts.iter().enumerate() {\n            if count >= 3 {\n                assert!(vals.contains(&val));\n            }\n        }\n    }\n\n    #[pg_test]\n    fn test_freq_agg_rollup_maintains_invariant() {\n        // The frequency agg invariant is that any element with frequency >= f will appear in the freq_agg(f)\n\n        // This test will randomly generate 200 values in the uniform range [0, 99] and check to see any value\n        // that shows up at least 3 times appears in a frequency aggregate created with freq = 0.015\n        let rand100 = Uniform::new_inclusive(0, 99);\n        let mut rng = rand::thread_rng();\n\n        let mut counts = [0; 100];\n\n        let freq = 0.015;\n        let fcinfo = std::ptr::null_mut(); // dummy value, will use default collation\n\n        let mut aggs = vec![];\n        for _ in 0..4 {\n            let mut state = None.into();\n            for _ in 0..50 {\n                let v = rand100.sample(&mut rng);\n                let value = unsafe {\n                    AnyElement::from_polymorphic_datum(\n                        pg_sys::Datum::from(v),\n                        false,\n                        pg_sys::INT4OID,\n                    )\n                };\n                state = super::freq_agg_trans(state, freq, value, fcinfo).unwrap();\n                counts[v] += 1;\n            }\n            aggs.push(space_saving_final(state, fcinfo).unwrap());\n        }\n\n        let state = {\n            let mut state = None.into();\n            for agg in aggs {\n                state = super::rollup_agg_trans(state, Some(agg), fcinfo).unwrap();\n            }\n            space_saving_final(state, fcinfo).unwrap()\n        };\n        let vals: std::collections::HashSet<usize> =\n            state.datums.iter().map(|datum| datum.value()).collect();\n\n        for (val, &count) in counts.iter().enumerate() {\n            if count >= 3 {\n                assert!(vals.contains(&val));\n            }\n        }\n    }\n\n    #[pg_test]\n    fn test_mcv_agg_invariant() {\n        // The ton agg invariant is that we'll be able to track the top n values for any data\n        // with a distribution at least as skewed as a zeta distribution\n\n        // To test this we will generate a mcv aggregate with a random skew (1.01 - 2.0) and\n        // n (5-10).  We then generate a random sample with skew 5% greater than our aggregate\n        // (this should be enough to keep the sample above the target even with bad luck), and\n        // verify that we correctly identify the top n values.\n        let mut rng = rand::thread_rng();\n\n        let n = rng.next_u64() % 6 + 5;\n        let skew = (rng.next_u64() % 100) as f64 / 100. + 1.01;\n\n        let zeta = Zeta::new(skew * 1.05).unwrap();\n\n        let mut counts = [0; 100];\n\n        let mut state = None.into();\n        let fcinfo = std::ptr::null_mut(); // dummy value, will use default collation\n\n        for _ in 0..100000 {\n            let v = zeta.sample(&mut rng).floor() as usize;\n            if v == usize::MAX {\n                continue; // These tail values can start to add up at low skew values\n            }\n            let value = unsafe {\n                AnyElement::from_polymorphic_datum(pg_sys::Datum::from(v), false, pg_sys::INT4OID)\n            };\n            state = super::mcv_agg_with_skew_trans(state, n as i32, skew, value, fcinfo).unwrap();\n            if v < 100 {\n                // anything greater than 100 will not be in the top values\n                counts[v] += 1;\n            }\n        }\n\n        let state = space_saving_final(state, fcinfo).unwrap();\n        let value =\n            unsafe { AnyElement::from_polymorphic_datum(Datum::from(0), false, pg_sys::INT4OID) };\n        let t: Vec<AnyElement> = default_topn(state, Some(value.unwrap())).collect();\n        let agg_topn: Vec<usize> = t.iter().map(|x| x.datum().value()).collect();\n\n        let mut temp: Vec<(usize, &usize)> = counts.iter().enumerate().collect();\n        temp.sort_by(|(_, cnt1), (_, cnt2)| cnt2.cmp(cnt1)); // descending order by count\n        let top_vals: Vec<usize> = temp.into_iter().map(|(val, _)| val).collect();\n\n        for i in 0..n as usize {\n            assert_eq!(agg_topn[i], top_vals[i]);\n        }\n    }\n}\n"
  },
  {
    "path": "extension/src/gauge_agg.rs",
    "content": "use pgrx::*;\n\nuse serde::{Deserialize, Serialize};\n\nuse counter_agg::{range::I64Range, GaugeSummaryBuilder, MetricSummary};\nuse flat_serialize_macro::FlatSerializable;\nuse stats_agg::stats2d::StatsSummary2D;\nuse tspoint::TSPoint;\n\nuse crate::{\n    accessors::{\n        AccessorCorr, AccessorCounterZeroTime, AccessorDelta, AccessorExtrapolatedDelta,\n        AccessorExtrapolatedRate, AccessorIdeltaLeft, AccessorIdeltaRight, AccessorIntercept,\n        AccessorIrateLeft, AccessorIrateRight, AccessorNumChanges, AccessorNumElements,\n        AccessorRate, AccessorSlope, AccessorTimeDelta, AccessorWithBounds,\n    },\n    aggregate_utils::in_aggregate_context,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    range::{get_range, I64RangeWrapper},\n    raw::{bytea, tstzrange},\n    ron_inout_funcs,\n};\n\n// TODO move to share with counter_agg\n#[derive(Clone, Debug, PartialEq, Deserialize, Serialize, FlatSerializable)]\n#[repr(C)]\npub struct FlatSummary {\n    stats: StatsSummary2D<f64>,\n    first: TSPoint,\n    second: TSPoint,\n    penultimate: TSPoint,\n    last: TSPoint,\n    reset_sum: f64,\n    num_resets: u64,\n    num_changes: u64,\n    bounds: I64RangeWrapper,\n}\n\n#[pg_schema]\nmod toolkit_experimental {\n    use super::*;\n\n    pg_type! {\n        #[derive(Debug, PartialEq)]\n        struct GaugeSummary {\n            #[flat_serialize::flatten]\n            summary: FlatSummary,\n        }\n    }\n\n    impl GaugeSummary {\n        pub(super) fn interpolate(\n            &self,\n            interval_start: i64,\n            interval_len: i64,\n            prev: Option<GaugeSummary>,\n            next: Option<GaugeSummary>,\n        ) -> GaugeSummary {\n            let this = MetricSummary::from(self.clone());\n            let prev = prev.map(MetricSummary::from);\n            let next = next.map(MetricSummary::from);\n\n            let prev = if this.first.ts > interval_start {\n                prev.map(|summary| {\n                    time_weighted_average::TimeWeightMethod::Linear\n                        .interpolate(summary.last, Some(this.first), interval_start)\n                        .expect(\"unable to interpolate lower bound\")\n                })\n            } else {\n                None\n            };\n\n            let next = next.map(|summary| {\n                time_weighted_average::TimeWeightMethod::Linear\n                    .interpolate(\n                        this.last,\n                        Some(summary.first),\n                        interval_start + interval_len,\n                    )\n                    .expect(\"unable to interpolate upper bound\")\n            });\n\n            let builder = prev.map(|pt| GaugeSummaryBuilder::new(&pt, None));\n            let mut builder = builder.map_or_else(\n                || {\n                    let mut summary = this.clone();\n                    summary.bounds = None;\n                    summary.into()\n                },\n                |mut builder| {\n                    builder\n                        .combine(&this)\n                        .expect(\"unable to add data to interpolation\");\n                    builder\n                },\n            );\n\n            if let Some(next) = next {\n                builder\n                    .add_point(&next)\n                    .expect(\"unable to add final interpolated point\");\n            }\n\n            builder.build().into()\n        }\n    }\n\n    ron_inout_funcs!(GaugeSummary);\n}\n\nuse toolkit_experimental::*;\n\n// TODO reunify with crate::counter_agg::CounterSummaryTransSate\n// TODO move to crate::metrics::TransState (taking FnOnce()->MetricSummaryBuilder to support both)\n#[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]\nstruct GaugeSummaryTransState {\n    #[serde(skip)]\n    point_buffer: Vec<TSPoint>,\n    #[serde(skip)]\n    bounds: Option<I64Range>, // stores bounds until we combine points, after which, the bounds are stored in each summary\n    // We have a summary buffer here in order to deal with the fact that when the cmobine function gets called it\n    // must first build up a buffer of InternalMetricSummaries, then sort them, then call the combine function in\n    // the correct order.\n    summary_buffer: Vec<MetricSummary>,\n}\n\nimpl GaugeSummaryTransState {\n    fn new() -> Self {\n        Self {\n            point_buffer: vec![],\n            bounds: None,\n            summary_buffer: vec![],\n        }\n    }\n\n    fn push_point(&mut self, value: TSPoint) {\n        self.point_buffer.push(value);\n    }\n\n    fn combine_points(&mut self) {\n        if self.point_buffer.is_empty() {\n            return;\n        }\n        self.point_buffer.sort_unstable_by_key(|p| p.ts);\n        let mut iter = self.point_buffer.iter();\n        let mut summary = GaugeSummaryBuilder::new(iter.next().unwrap(), self.bounds);\n        for p in iter {\n            summary\n                .add_point(p)\n                .unwrap_or_else(|e| pgrx::error!(\"{}\", e));\n        }\n        self.point_buffer.clear();\n        // TODO build method should check validity\n        // check bounds only after we've combined all the points, so we aren't doing it all the time.\n        if !summary.bounds_valid() {\n            panic!(\"Metric bounds invalid\")\n        }\n        self.summary_buffer.push(summary.build());\n    }\n\n    fn push_summary(&mut self, other: &Self) {\n        let sum_iter = other.summary_buffer.iter();\n        for sum in sum_iter {\n            self.summary_buffer.push(sum.clone());\n        }\n    }\n\n    fn combine_summaries(&mut self) {\n        self.combine_points();\n\n        if self.summary_buffer.len() <= 1 {\n            return;\n        }\n        self.summary_buffer.sort_unstable_by_key(|s| s.first.ts);\n        let mut sum_iter = self.summary_buffer.drain(..);\n        let first = sum_iter.next().expect(\"already handled empty case\");\n        let mut new_summary = GaugeSummaryBuilder::from(first);\n        for sum in sum_iter {\n            new_summary\n                .combine(&sum)\n                .unwrap_or_else(|e| pgrx::error!(\"{}\", e));\n        }\n        self.summary_buffer.push(new_summary.build());\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, strict, schema = \"toolkit_experimental\")]\nfn gauge_summary_trans_serialize(state: Internal) -> bytea {\n    let mut state = state;\n    let state: &mut GaugeSummaryTransState = unsafe { state.get_mut().unwrap() };\n    state.combine_summaries();\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_summary_trans_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    gauge_summary_trans_deserialize_inner(bytes).internal()\n}\nfn gauge_summary_trans_deserialize_inner(bytes: bytea) -> Inner<GaugeSummaryTransState> {\n    let c: GaugeSummaryTransState = crate::do_deserialize!(bytes, GaugeSummaryTransState);\n    c.into()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_agg_trans(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    bounds: Option<tstzrange>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    gauge_agg_trans_inner(unsafe { state.to_inner() }, ts, val, bounds, fcinfo).internal()\n}\nfn gauge_agg_trans_inner(\n    state: Option<Inner<GaugeSummaryTransState>>,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    bounds: Option<tstzrange>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<GaugeSummaryTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let p = match (ts, val) {\n                (_, None) => return state,\n                (None, _) => return state,\n                (Some(ts), Some(val)) => TSPoint { ts: ts.into(), val },\n            };\n            match state {\n                None => {\n                    let mut s = GaugeSummaryTransState::new();\n                    if let Some(r) = bounds {\n                        s.bounds = get_range(r.0.cast_mut_ptr());\n                    }\n                    s.push_point(p);\n                    Some(s.into())\n                }\n                Some(mut s) => {\n                    s.push_point(p);\n                    Some(s)\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_agg_trans_no_bounds(\n    state: Internal,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    gauge_agg_trans_inner(unsafe { state.to_inner() }, ts, val, None, fcinfo).internal()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_agg_summary_trans(\n    state: Internal,\n    value: Option<GaugeSummary>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    gauge_agg_summary_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\nfn gauge_agg_summary_trans_inner(\n    state: Option<Inner<GaugeSummaryTransState>>,\n    value: Option<GaugeSummary>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<GaugeSummaryTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, value) {\n            (state, None) => state,\n            (None, Some(value)) => {\n                let mut state = GaugeSummaryTransState::new();\n                state.summary_buffer.push(value.into());\n                Some(state.into())\n            }\n            (Some(mut state), Some(value)) => {\n                state.summary_buffer.push(value.into());\n                Some(state)\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_agg_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { gauge_agg_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\nfn gauge_agg_combine_inner(\n    state1: Option<Inner<GaugeSummaryTransState>>,\n    state2: Option<Inner<GaugeSummaryTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<GaugeSummaryTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state1, state2) {\n                (None, None) => None,\n                (None, Some(state2)) => {\n                    let mut s = state2.clone();\n                    s.combine_points();\n                    Some(s.into())\n                }\n                (Some(state1), None) => {\n                    let mut s = state1.clone();\n                    s.combine_points();\n                    Some(s.into())\n                } //should I make these return themselves?\n                (Some(state1), Some(state2)) => {\n                    let mut s1 = state1.clone(); // is there a way to avoid if it doesn't need it?\n                    s1.combine_points();\n                    let mut s2 = state2.clone();\n                    s2.combine_points();\n                    s2.push_summary(&s1);\n                    Some(s2.into())\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_agg_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<GaugeSummary> {\n    gauge_agg_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\nfn gauge_agg_final_inner(\n    state: Option<Inner<GaugeSummaryTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<GaugeSummary> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state.clone(),\n            };\n            state.combine_summaries();\n            debug_assert!(state.summary_buffer.len() <= 1);\n            match state.summary_buffer.pop() {\n                None => None,\n                Some(st) => {\n                    // there are some edge cases that this should prevent, but I'm not sure it's necessary, we do check the bounds in the functions that use them.\n                    if !st.bounds_valid() {\n                        panic!(\"Metric bounds invalid\")\n                    }\n                    Some(GaugeSummary::from(st))\n                }\n            }\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.gauge_agg( ts timestamptz, value DOUBLE PRECISION, bounds tstzrange )\\n\\\n    (\\n\\\n        sfunc = toolkit_experimental.gauge_agg_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = toolkit_experimental.gauge_agg_final,\\n\\\n        combinefunc = toolkit_experimental.gauge_agg_combine,\\n\\\n        serialfunc = toolkit_experimental.gauge_summary_trans_serialize,\\n\\\n        deserialfunc = toolkit_experimental.gauge_summary_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\",\n    name = \"gauge_agg\",\n    requires = [\n        gauge_agg_trans,\n        gauge_agg_final,\n        gauge_agg_combine,\n        gauge_summary_trans_serialize,\n        gauge_summary_trans_deserialize\n    ],\n);\n\n// allow calling gauge agg without bounds provided.\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.gauge_agg( ts timestamptz, value DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = toolkit_experimental.gauge_agg_trans_no_bounds,\\n\\\n        stype = internal,\\n\\\n        finalfunc = toolkit_experimental.gauge_agg_final,\\n\\\n        combinefunc = toolkit_experimental.gauge_agg_combine,\\n\\\n        serialfunc = toolkit_experimental.gauge_summary_trans_serialize,\\n\\\n        deserialfunc = toolkit_experimental.gauge_summary_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\\\n\",\n    name = \"gauge_agg2\",\n    requires = [\n        gauge_agg_trans_no_bounds,\n        gauge_agg_final,\n        gauge_agg_combine,\n        gauge_summary_trans_serialize,\n        gauge_summary_trans_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.rollup(gs toolkit_experimental.GaugeSummary)\\n\\\n    (\\n\\\n        sfunc = toolkit_experimental.gauge_agg_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = toolkit_experimental.gauge_agg_final,\\n\\\n        combinefunc = toolkit_experimental.gauge_agg_combine,\\n\\\n        serialfunc = toolkit_experimental.gauge_summary_trans_serialize,\\n\\\n        deserialfunc = toolkit_experimental.gauge_summary_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\\\n\",\n    name = \"gauge_rollup\",\n    requires = [\n        gauge_agg_summary_trans,\n        gauge_agg_final,\n        gauge_agg_combine,\n        gauge_summary_trans_serialize,\n        gauge_summary_trans_deserialize\n    ],\n);\n\n// TODO Reconsider using the same pg_type for counter and gauge aggregates to avoid duplicating all these functions.\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_delta(sketch: GaugeSummary, _accessor: AccessorDelta) -> f64 {\n    delta(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn delta(summary: GaugeSummary) -> f64 {\n    MetricSummary::from(summary).delta()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_gauge_agg_rate(sketch: GaugeSummary, _accessor: AccessorRate) -> Option<f64> {\n    rate(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn rate(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).rate()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_time_delta(sketch: GaugeSummary, _accessor: AccessorTimeDelta) -> f64 {\n    time_delta(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn time_delta(summary: GaugeSummary) -> f64 {\n    MetricSummary::from(summary).time_delta()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_irate_left(sketch: GaugeSummary, _accessor: AccessorIrateLeft) -> Option<f64> {\n    irate_left(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn irate_left(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).irate_left()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_irate_right(sketch: GaugeSummary, _accessor: AccessorIrateRight) -> Option<f64> {\n    irate_right(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn irate_right(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).irate_right()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_idelta_left(sketch: GaugeSummary, _accessor: AccessorIdeltaLeft) -> f64 {\n    idelta_left(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn idelta_left(summary: GaugeSummary) -> f64 {\n    MetricSummary::from(summary).idelta_left()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_idelta_right(sketch: GaugeSummary, _accessor: AccessorIdeltaRight) -> f64 {\n    idelta_right(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn idelta_right(summary: GaugeSummary) -> f64 {\n    MetricSummary::from(summary).idelta_right()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_with_bounds(sketch: GaugeSummary, accessor: AccessorWithBounds) -> GaugeSummary {\n    let mut builder = GaugeSummaryBuilder::from(MetricSummary::from(sketch));\n    builder.set_bounds(accessor.bounds());\n    builder.build().into()\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn with_bounds(summary: GaugeSummary, bounds: tstzrange) -> GaugeSummary {\n    // TODO dedup with previous by using apply_bounds\n    unsafe {\n        let ptr = bounds.0.cast_mut_ptr();\n        let mut builder = GaugeSummaryBuilder::from(MetricSummary::from(summary));\n        builder.set_bounds(get_range(ptr));\n        builder.build().into()\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_extrapolated_delta(\n    sketch: GaugeSummary,\n    _accessor: AccessorExtrapolatedDelta,\n) -> Option<f64> {\n    extrapolated_delta(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn extrapolated_delta(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).prometheus_delta().unwrap()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn interpolated_delta(\n    summary: GaugeSummary,\n    start: crate::raw::TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<GaugeSummary>,\n    next: Option<GaugeSummary>,\n) -> f64 {\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    MetricSummary::from(summary.interpolate(start.into(), interval, prev, next)).delta()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_extrapolated_rate(\n    sketch: GaugeSummary,\n    _accessor: AccessorExtrapolatedRate,\n) -> Option<f64> {\n    extrapolated_rate(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn extrapolated_rate(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).prometheus_rate().unwrap()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn interpolated_rate(\n    summary: GaugeSummary,\n    start: crate::raw::TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<GaugeSummary>,\n    next: Option<GaugeSummary>,\n) -> Option<f64> {\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    MetricSummary::from(summary.interpolate(start.into(), interval, prev, next)).rate()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_num_elements(sketch: GaugeSummary, _accessor: AccessorNumElements) -> i64 {\n    num_elements(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn num_elements(summary: GaugeSummary) -> i64 {\n    MetricSummary::from(summary).stats.n as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_num_changes(sketch: GaugeSummary, _accessor: AccessorNumChanges) -> i64 {\n    num_changes(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn num_changes(summary: GaugeSummary) -> i64 {\n    MetricSummary::from(summary).num_changes as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_slope(sketch: GaugeSummary, _accessor: AccessorSlope) -> Option<f64> {\n    slope(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn slope(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).stats.slope()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_intercept(sketch: GaugeSummary, _accessor: AccessorIntercept) -> Option<f64> {\n    intercept(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn intercept(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).stats.intercept()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_corr(sketch: GaugeSummary, _accessor: AccessorCorr) -> Option<f64> {\n    corr(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn corr(summary: GaugeSummary) -> Option<f64> {\n    MetricSummary::from(summary).stats.corr()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\nfn arrow_zero_time(\n    sketch: GaugeSummary,\n    __accessor: AccessorCounterZeroTime,\n) -> Option<crate::raw::TimestampTz> {\n    gauge_zero_time(sketch)\n}\n\n#[pg_extern(strict, immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn gauge_zero_time(summary: GaugeSummary) -> Option<crate::raw::TimestampTz> {\n    Some(((MetricSummary::from(summary).stats.x_intercept()? * 1_000_000.0) as i64).into())\n}\n\nimpl From<GaugeSummary> for MetricSummary {\n    fn from(pg: GaugeSummary) -> Self {\n        Self {\n            first: pg.summary.first,\n            second: pg.summary.second,\n            penultimate: pg.summary.penultimate,\n            last: pg.summary.last,\n            reset_sum: pg.summary.reset_sum,\n            num_resets: pg.summary.num_resets,\n            num_changes: pg.summary.num_changes,\n            stats: pg.summary.stats,\n            bounds: pg.summary.bounds.to_i64range(),\n        }\n    }\n}\n\nimpl From<MetricSummary> for GaugeSummary {\n    fn from(internal: MetricSummary) -> Self {\n        unsafe {\n            flatten!(GaugeSummary {\n                summary: FlatSummary {\n                    stats: internal.stats,\n                    first: internal.first,\n                    second: internal.second,\n                    penultimate: internal.penultimate,\n                    last: internal.last,\n                    reset_sum: internal.reset_sum,\n                    num_resets: internal.num_resets,\n                    num_changes: internal.num_changes,\n                    bounds: I64RangeWrapper::from_i64range(internal.bounds)\n                }\n            })\n        }\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx_macros::pg_test;\n\n    use crate::counter_agg::testing::*;\n\n    use super::*;\n\n    macro_rules! select_one {\n        ($client:expr, $stmt:expr, $type:ty) => {\n            $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_one::<$type>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    #[pg_test]\n    fn round_trip() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"SET TIME ZONE 'UTC'\", None, &[]).unwrap();\n            let stmt = \"INSERT INTO test VALUES\\\n                ('2020-01-01 00:00:00+00', 10.0),\\\n                ('2020-01-01 00:01:00+00', 20.0),\\\n                ('2020-01-01 00:02:00+00', 30.0),\\\n                ('2020-01-01 00:03:00+00', 20.0),\\\n                ('2020-01-01 00:04:00+00', 10.0),\\\n                ('2020-01-01 00:05:00+00', 20.0),\\\n                ('2020-01-01 00:06:00+00', 10.0),\\\n                ('2020-01-01 00:07:00+00', 30.0),\\\n                ('2020-01-01 00:08:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                summary:(\\\n                    stats:(\\\n                        n:9,\\\n                        sx:5680370160,\\\n                        sx2:216000,\\\n                        sx3:0,\\\n                        sx4:9175680000,\\\n                        sy:160,\\\n                        sy2:555.5555555555555,\\\n                        sy3:1802.4691358024695,\\\n                        sy4:59341.563786008235,\\\n                        sxy:-600\\\n                    ),\\\n                    first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                    second:(ts:\\\"2020-01-01 00:01:00+00\\\",val:20),\\\n                    penultimate:(ts:\\\"2020-01-01 00:07:00+00\\\",val:30),\\\n                    last:(ts:\\\"2020-01-01 00:08:00+00\\\",val:10),\\\n                    reset_sum:0,\\\n                    num_resets:0,\\\n                    num_changes:8,\\\n                    bounds:(\\\n                        is_present:0,\\\n                        has_left:0,\\\n                        has_right:0,\\\n                        padding:(0,0,0,0,0),\\\n                        left:None,\\\n                        right:None\\\n                    )\\\n                )\\\n            )\";\n\n            assert_eq!(\n                expected,\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.gauge_agg(ts, val)::TEXT FROM test\",\n                    String\n                )\n            );\n\n            assert_eq!(\n                expected,\n                select_one!(\n                    client,\n                    &format!(\"SELECT '{expected}'::toolkit_experimental.GaugeSummary::TEXT\"),\n                    String\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_gauge_decrease() {\n        Spi::connect_mut(|client| {\n            decrease(client);\n            let stmt = \"SELECT toolkit_experimental.delta(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(-20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_gauge_increase() {\n        Spi::connect_mut(|client| {\n            increase(client);\n            let stmt = \"SELECT toolkit_experimental.delta(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_gauge_decrease_then_increase_to_same_value() {\n        Spi::connect_mut(|client| {\n            decrease_then_increase_to_same_value(client);\n            let stmt = \"SELECT toolkit_experimental.delta(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(0.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn delta_after_gauge_increase_then_decrease_to_same_value() {\n        Spi::connect_mut(|client| {\n            increase_then_decrease_to_same_value(client);\n            let stmt = \"SELECT toolkit_experimental.delta(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(0.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_gauge_decrease() {\n        Spi::connect_mut(|client| {\n            decrease(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_left(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_gauge_increase() {\n        Spi::connect_mut(|client| {\n            increase(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_left(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_gauge_increase_then_decrease_to_same_value() {\n        Spi::connect_mut(|client| {\n            increase_then_decrease_to_same_value(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_left(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_left_after_gauge_decrease_then_increase_to_same_value() {\n        Spi::connect_mut(|client| {\n            decrease_then_increase_to_same_value(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_left(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_gauge_decrease() {\n        Spi::connect_mut(|client| {\n            decrease(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_right(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_gauge_increase() {\n        Spi::connect_mut(|client| {\n            increase(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_right(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_gauge_increase_then_decrease_to_same_value() {\n        Spi::connect_mut(|client| {\n            increase_then_decrease_to_same_value(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_right(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(10.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    #[pg_test]\n    fn idelta_right_after_gauge_decrease_then_increase_to_same_value() {\n        Spi::connect_mut(|client| {\n            decrease_then_increase_to_same_value(client);\n            let stmt = \"SELECT toolkit_experimental.idelta_right(toolkit_experimental.gauge_agg(ts, val)) FROM test\";\n            assert_eq!(20.0, select_one!(client, stmt, f64));\n        });\n    }\n\n    // TODO 3rd copy of this...\n    #[track_caller]\n    fn assert_close_enough(p1: &MetricSummary, p2: &MetricSummary) {\n        assert_eq!(p1.first, p2.first, \"first\");\n        assert_eq!(p1.second, p2.second, \"second\");\n        assert_eq!(p1.penultimate, p2.penultimate, \"penultimate\");\n        assert_eq!(p1.last, p2.last, \"last\");\n        assert_eq!(p1.num_changes, p2.num_changes, \"num_changes\");\n        assert_eq!(p1.num_resets, p2.num_resets, \"num_resets\");\n        assert_eq!(p1.stats.n, p2.stats.n, \"n\");\n        use approx::assert_relative_eq;\n        assert_relative_eq!(p1.stats.sx, p2.stats.sx);\n        assert_relative_eq!(p1.stats.sx2, p2.stats.sx2);\n        assert_relative_eq!(p1.stats.sy, p2.stats.sy);\n        assert_relative_eq!(p1.stats.sy2, p2.stats.sy2);\n        assert_relative_eq!(p1.stats.sxy, p2.stats.sxy);\n    }\n\n    #[pg_test]\n    fn rollup() {\n        Spi::connect_mut(|client| {\n            // needed so that GaugeSummary type can be resolved\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // This tests GaugeSummary::single_value - the old first == last\n            // check erroneously saw 21.0 == 21.0 and called it a single value.\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:00:00+00', 10.0), ('2020-01-01 00:01:00+00', 21.0), ('2020-01-01 00:01:00+00', 22.0), ('2020-01-01 00:01:00+00', 21.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:02:00+00', 10.0), ('2020-01-01 00:03:00+00', 20.0), ('2020-01-01 00:04:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:08:00+00', 30.0), ('2020-01-01 00:10:00+00', 30.0), ('2020-01-01 00:10:30+00', 10.0), ('2020-01-01 00:20:00+00', 40.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            //combine function works as expected\n            let stmt = \"SELECT gauge_agg(ts, val) FROM test\";\n            let a = select_one!(client, stmt, GaugeSummary);\n            let stmt = \"WITH t as (SELECT date_trunc('minute', ts), gauge_agg(ts, val) as agg FROM test group by 1 ) SELECT rollup(agg) FROM t\";\n            let b = select_one!(client, stmt, GaugeSummary);\n            assert_close_enough(&a.into(), &b.into());\n        });\n    }\n\n    #[pg_test]\n    fn gauge_agg_interpolation() {\n        Spi::connect_mut(|client| {\n            client.update(\n                \"CREATE TABLE test(time timestamptz, value double precision, bucket timestamptz)\",\n                None,\n                &[]\n            ).unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                ('2020-1-1 10:00'::timestamptz, 10.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 12:00'::timestamptz, 40.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 16:00'::timestamptz, 20.0, '2020-1-1'::timestamptz),\n                ('2020-1-2 2:00'::timestamptz, 15.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 12:00'::timestamptz, 50.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 20:00'::timestamptz, 25.0, '2020-1-2'::timestamptz),\n                ('2020-1-3 4:00'::timestamptz, 30.0, '2020-1-3'::timestamptz),\n                ('2020-1-3 12:00'::timestamptz, 0.0, '2020-1-3'::timestamptz), \n                ('2020-1-3 16:00'::timestamptz, 35.0, '2020-1-3'::timestamptz)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut deltas = client\n                .update(\n                    r#\"SELECT\n                toolkit_experimental.interpolated_delta(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, toolkit_experimental.gauge_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, start at 10, interpolated end of day is 16\n            assert_eq!(deltas.next().unwrap()[1].value().unwrap(), Some(16. - 10.));\n            // Day 2, interpolated start is 16, interpolated end is 27.5\n            assert_eq!(deltas.next().unwrap()[1].value().unwrap(), Some(27.5 - 16.));\n            // Day 3, interpolated start is 27.5, end is 35\n            assert_eq!(deltas.next().unwrap()[1].value().unwrap(), Some(35. - 27.5));\n\n            let mut rates = client\n                .update(\n                    r#\"SELECT\n                toolkit_experimental.interpolated_rate(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, toolkit_experimental.gauge_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, 14 hours (rate is per second)\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((16. - 10.) / (14. * 60. * 60.))\n            );\n            // Day 2, 24 hours\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((27.5 - 16.) / (24. * 60. * 60.))\n            );\n            // Day 3, 16 hours\n            assert_eq!(\n                rates.next().unwrap()[1].value().unwrap(),\n                Some((35. - 27.5) / (16. * 60. * 60.))\n            );\n        });\n    }\n\n    #[pg_test]\n    fn guage_agg_interpolated_delta_with_aligned_point() {\n        Spi::connect_mut(|client| {\n            client.update(\n                \"CREATE TABLE test(time timestamptz, value double precision, bucket timestamptz)\",\n                None,\n                &[]\n            ).unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                ('2020-1-1 10:00'::timestamptz, 10.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 12:00'::timestamptz, 40.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 16:00'::timestamptz, 20.0, '2020-1-1'::timestamptz),\n                ('2020-1-2 0:00'::timestamptz, 15.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 12:00'::timestamptz, 50.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 20:00'::timestamptz, 25.0, '2020-1-2'::timestamptz)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut deltas = client\n                .update(\n                    r#\"SELECT\n                toolkit_experimental.interpolated_delta(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket), \n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, toolkit_experimental.gauge_agg(time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            // Day 1, start at 10, interpolated end of day is 15 (after reset)\n            assert_eq!(deltas.next().unwrap()[1].value().unwrap(), Some(15. - 10.));\n            // Day 2, start is 15, end is 25\n            assert_eq!(deltas.next().unwrap()[1].value().unwrap(), Some(25. - 15.));\n            assert!(deltas.next().is_none());\n        });\n    }\n\n    #[pg_test]\n    fn no_results_on_null_input() {\n        Spi::connect_mut(|client| {\n            // needed so that GaugeSummary type can be resolved\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let stmt = \"INSERT INTO test VALUES (NULL, NULL)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"SELECT toolkit_experimental.gauge_agg(ts, val) FROM test\";\n            assert!(client\n                .update(stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_one::<GaugeSummary>()\n                .unwrap()\n                .is_none());\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/heartbeat_agg/accessors.rs",
    "content": "use pgrx::*;\n\nuse crate::{\n    flatten,\n    heartbeat_agg::{HeartbeatAgg, HeartbeatAggData},\n    pg_type, ron_inout_funcs,\n};\n\nfn empty_agg<'a>() -> HeartbeatAgg<'a> {\n    unsafe {\n        flatten!(HeartbeatAgg {\n            start_time: 0,\n            end_time: 0,\n            last_seen: 0,\n            interval_len: 0,\n            num_intervals: 0,\n            interval_starts: vec!().into(),\n            interval_ends: vec!().into(),\n        })\n    }\n}\n\npg_type! {\n    struct HeartbeatInterpolatedUptimeAccessor<'input> {\n        has_prev : u64,\n        prev : HeartbeatAggData<'input>,\n    }\n}\n\nron_inout_funcs!(HeartbeatInterpolatedUptimeAccessor<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_uptime\")]\nfn heartbeat_agg_interpolated_uptime_accessor<'a>(\n    prev: Option<HeartbeatAgg<'a>>,\n) -> HeartbeatInterpolatedUptimeAccessor<'a> {\n    let has_prev = u64::from(prev.is_some());\n    let prev = prev.unwrap_or_else(empty_agg).0;\n\n    crate::build! {\n        HeartbeatInterpolatedUptimeAccessor {\n            has_prev,\n            prev,\n        }\n    }\n}\n\nimpl<'a> HeartbeatInterpolatedUptimeAccessor<'a> {\n    pub fn pred(&self) -> Option<HeartbeatAgg<'a>> {\n        if self.has_prev == 0 {\n            None\n        } else {\n            Some(self.prev.clone().into())\n        }\n    }\n}\n\npg_type! {\n    struct HeartbeatInterpolatedDowntimeAccessor<'input> {\n        has_prev : u64,\n        prev : HeartbeatAggData<'input>,\n    }\n}\n\nron_inout_funcs!(HeartbeatInterpolatedDowntimeAccessor<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_downtime\")]\nfn heartbeat_agg_interpolated_downtime_accessor<'a>(\n    prev: Option<HeartbeatAgg<'a>>,\n) -> HeartbeatInterpolatedDowntimeAccessor<'a> {\n    let has_prev = u64::from(prev.is_some());\n    let prev = prev.unwrap_or_else(empty_agg).0;\n\n    crate::build! {\n        HeartbeatInterpolatedDowntimeAccessor {\n            has_prev,\n            prev,\n        }\n    }\n}\n\nimpl<'a> HeartbeatInterpolatedDowntimeAccessor<'a> {\n    pub fn pred(&self) -> Option<HeartbeatAgg<'a>> {\n        if self.has_prev == 0 {\n            None\n        } else {\n            Some(self.prev.clone().into())\n        }\n    }\n}\n\npg_type! {\n    struct HeartbeatInterpolateAccessor<'input> {\n        has_prev : u64,\n        prev : HeartbeatAggData<'input>,\n    }\n}\n\nron_inout_funcs!(HeartbeatInterpolateAccessor<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolate\")]\nfn heartbeat_agg_interpolate_accessor<'a>(\n    prev: Option<HeartbeatAgg<'a>>,\n) -> HeartbeatInterpolateAccessor<'a> {\n    let has_prev = u64::from(prev.is_some());\n    let prev = prev.unwrap_or_else(empty_agg).0;\n\n    crate::build! {\n        HeartbeatInterpolateAccessor {\n            has_prev,\n            prev,\n        }\n    }\n}\n\nimpl<'a> HeartbeatInterpolateAccessor<'a> {\n    pub fn pred(&self) -> Option<HeartbeatAgg<'a>> {\n        if self.has_prev == 0 {\n            None\n        } else {\n            Some(self.prev.clone().into())\n        }\n    }\n}\n\npg_type! {\n    struct HeartbeatTrimToAccessor {\n        start : i64,\n        end : i64,\n    }\n}\n\nron_inout_funcs!(HeartbeatTrimToAccessor);\n\n// Note that this is unable to take only a duration, as we don't have the functionality to store\n// an interval in PG format and are unable to convert it to an int without a reference time.\n// This is a difference from the inline function.\n#[pg_extern(immutable, parallel_safe, name = \"trim_to\")]\nfn heartbeat_agg_trim_to_accessor(\n    start: crate::raw::TimestampTz,\n    duration: default!(Option<crate::raw::Interval>, \"NULL\"),\n) -> HeartbeatTrimToAccessor {\n    let end = duration\n        .map(|intv| crate::datum_utils::ts_interval_sum_to_ms(&start, &intv))\n        .unwrap_or(0);\n    let start = i64::from(start);\n\n    crate::build! {\n        HeartbeatTrimToAccessor {\n            start,\n            end,\n        }\n    }\n}\n"
  },
  {
    "path": "extension/src/heartbeat_agg.rs",
    "content": "use pgrx::iter::TableIterator;\nuse pgrx::*;\n\nuse crate::{\n    accessors::{\n        AccessorDeadRanges, AccessorDowntime, AccessorLiveAt, AccessorLiveRanges, AccessorNumGaps,\n        AccessorNumLiveRanges, AccessorUptime,\n    },\n    aggregate_utils::in_aggregate_context,\n    datum_utils::interval_to_ms,\n    flatten,\n    palloc::{Inner, InternalAsValue, ToInternal},\n    pg_type,\n    raw::{Interval, TimestampTz},\n    ron_inout_funcs,\n};\n\nuse std::cmp::{max, min};\n\nmod accessors;\n\nuse accessors::{\n    HeartbeatInterpolateAccessor, HeartbeatInterpolatedDowntimeAccessor,\n    HeartbeatInterpolatedUptimeAccessor, HeartbeatTrimToAccessor,\n};\n\nconst BUFFER_SIZE: usize = 1000; // How many values to absorb before consolidating\n\n// Given the lack of a good range map class, or efficient predecessor operation on btrees,\n// the trans state will simply collect points and then process them in batches\npub struct HeartbeatTransState {\n    start: i64,\n    end: i64,\n    last: i64,\n    interval_len: i64,\n    buffer: Vec<i64>,\n    liveness: Vec<(i64, i64)>, // sorted array of non-overlapping (start_time, end_time)\n}\n\nimpl HeartbeatTransState {\n    pub fn new(start: i64, end: i64, interval: i64) -> Self {\n        assert!(end - start > interval, \"all points passed to heartbeat agg must occur in the 'agg_duration' interval after 'agg_start'\");\n        HeartbeatTransState {\n            start,\n            end,\n            last: i64::MIN,\n            interval_len: interval,\n            buffer: vec![],\n            liveness: vec![],\n        }\n    }\n\n    pub fn insert(&mut self, time: i64) {\n        assert!(time >= self.start && time < self.end, \"all points passed to heartbeat agg must occur in the 'agg_duration' interval after 'agg_start'\");\n        if self.buffer.len() >= BUFFER_SIZE {\n            self.process_batch();\n        }\n        self.buffer.push(time);\n    }\n\n    pub fn process_batch(&mut self) {\n        if self.buffer.is_empty() {\n            return;\n        }\n        self.buffer.sort_unstable();\n\n        if self.last < *self.buffer.last().unwrap() {\n            self.last = *self.buffer.last().unwrap();\n        }\n\n        let mut new_intervals = vec![];\n\n        let mut start = *self.buffer.first().unwrap();\n        let mut bound = start + self.interval_len;\n\n        for heartbeat in std::mem::take(&mut self.buffer).into_iter() {\n            if heartbeat <= bound {\n                bound = heartbeat + self.interval_len;\n            } else {\n                new_intervals.push((start, bound));\n                start = heartbeat;\n                bound = start + self.interval_len;\n            }\n        }\n        new_intervals.push((start, bound));\n\n        if self.liveness.is_empty() {\n            std::mem::swap(&mut self.liveness, &mut new_intervals);\n        } else {\n            self.combine_intervals(new_intervals)\n        }\n    }\n\n    // In general we shouldn't need to change these creation time parameters, but if\n    // we're combining with another interval this may be necessary.\n    fn extend_covered_interval(&mut self, new_start: i64, new_end: i64) {\n        assert!(new_start <= self.start && new_end >= self.end); // this is guaranteed by the combine function\n        self.start = new_start;\n\n        // extend last range if able\n        if self.end < new_end && self.last + self.interval_len > self.end {\n            assert!(!self.liveness.is_empty()); // above condition should be impossible without liveness data\n\n            let last_interval = self.liveness.last_mut().unwrap();\n            last_interval.1 = min(self.last + self.interval_len, new_end);\n        }\n        self.end = new_end;\n    }\n\n    fn combine_intervals(&mut self, new_intervals: Vec<(i64, i64)>) {\n        // Optimized path for nonoverlapping, ordered inputs\n        if self.last < new_intervals.first().unwrap().0 {\n            let mut new_intervals = new_intervals.into_iter();\n\n            // Grab the first new interval to check for overlap with the existing data\n            let first_new = new_intervals.next().unwrap();\n\n            if self.liveness.last().unwrap().1 >= first_new.0 {\n                // Note that the bound of the new interval must be >= the old bound\n                self.liveness.last_mut().unwrap().1 = first_new.1;\n            } else {\n                self.liveness.push(first_new);\n            }\n\n            for val in new_intervals {\n                self.liveness.push(val);\n            }\n            return;\n        }\n\n        let new_intervals = new_intervals.into_iter();\n        let old_intervals = std::mem::take(&mut self.liveness).into_iter();\n\n        // In the following while let block, test and control are used to track our two interval iterators.\n        // We will swap them back and forth to try to keep control as the iterator which has provided the current bound.\n        let mut test = new_intervals.peekable();\n        let mut control = old_intervals.peekable();\n\n        while let Some(interval) = if let Some((start1, _)) = control.peek() {\n            if let Some((start2, _)) = test.peek() {\n                let (start, mut bound) = if start1 < start2 {\n                    control.next().unwrap()\n                } else {\n                    std::mem::swap(&mut test, &mut control);\n                    control.next().unwrap()\n                };\n\n                while test.peek().is_some() && test.peek().unwrap().0 <= bound {\n                    let (_, new_bound) = test.next().unwrap();\n                    if new_bound > bound {\n                        std::mem::swap(&mut test, &mut control);\n                        bound = new_bound;\n                    }\n                }\n\n                Some((start, bound))\n            } else {\n                control.next()\n            }\n        } else {\n            test.next()\n        } {\n            self.liveness.push(interval)\n        }\n    }\n\n    pub fn combine(&mut self, mut other: HeartbeatTransState) {\n        assert!(self.interval_len == other.interval_len); // Nicer error would be nice here\n        self.process_batch();\n        other.process_batch();\n\n        let min_start = min(self.start, other.start);\n        let max_end = max(self.end, other.end);\n        self.extend_covered_interval(min_start, max_end);\n        other.extend_covered_interval(min_start, max_end);\n\n        self.combine_intervals(other.liveness);\n        self.last = max(self.last, other.last);\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\nimpl HeartbeatTransState {\n    pub fn get_buffer(&self) -> &Vec<i64> {\n        &self.buffer\n    }\n    pub fn get_liveness(&self) -> &Vec<(i64, i64)> {\n        &self.liveness\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct HeartbeatAgg<'input>\n    {\n        start_time : i64,\n        end_time : i64,\n        last_seen : i64,\n        interval_len : i64,\n        num_intervals : u64,\n        interval_starts : [i64; self.num_intervals],\n        interval_ends : [i64; self.num_intervals],\n    }\n}\n\nron_inout_funcs!(HeartbeatAgg<'input>);\n\nimpl HeartbeatAgg<'_> {\n    fn trim_to(self, start: Option<i64>, end: Option<i64>) -> HeartbeatAgg<'static> {\n        if (start.is_some() && start.unwrap() < self.start_time)\n            || (end.is_some() && end.unwrap() > self.end_time)\n        {\n            error!(\"Can not query beyond the original aggregate bounds\");\n        }\n\n        let mut starts: Vec<i64> = vec![];\n        let mut ends: Vec<i64> = vec![];\n        for i in 0..self.num_intervals as usize {\n            starts.push(self.interval_starts.slice()[i]);\n            ends.push(self.interval_ends.slice()[i]);\n        }\n\n        let low_idx = if let Some(start) = start {\n            let mut idx = 0;\n            while idx < self.num_intervals as usize && ends[idx] < start {\n                idx += 1;\n            }\n            if starts[idx] < start {\n                starts[idx] = start;\n            }\n            idx\n        } else {\n            0\n        };\n\n        let mut new_last = None;\n        let high_idx = if let Some(end) = end {\n            if self.num_intervals > 0 {\n                let mut idx = self.num_intervals as usize - 1;\n                while idx > low_idx && starts[idx] > end {\n                    idx -= 1;\n                }\n                new_last = Some(ends[idx] - self.interval_len);\n                if ends[idx] > end {\n                    if end < new_last.unwrap() {\n                        new_last = Some(end);\n                    }\n                    ends[idx] = end;\n                }\n                idx\n            } else {\n                self.num_intervals as usize - 1\n            }\n        } else {\n            self.num_intervals as usize - 1\n        };\n\n        unsafe {\n            flatten!(HeartbeatAgg {\n                start_time: start.unwrap_or(self.start_time),\n                end_time: end.unwrap_or(self.end_time),\n                last_seen: new_last.unwrap_or(self.last_seen),\n                interval_len: self.interval_len,\n                num_intervals: (high_idx - low_idx + 1) as u64,\n                interval_starts: starts[low_idx..=high_idx].into(),\n                interval_ends: ends[low_idx..=high_idx].into(),\n            })\n        }\n    }\n\n    fn sum_live_intervals(self) -> i64 {\n        let starts = self.interval_starts.as_slice();\n        let ends = self.interval_ends.as_slice();\n        let mut sum = 0;\n        for i in 0..self.num_intervals as usize {\n            sum += ends[i] - starts[i];\n        }\n        sum\n    }\n\n    fn interpolate_start(&mut self, pred: &Self) {\n        // only allow interpolation of non-overlapping ranges\n        assert!(pred.end_time <= self.start_time);\n        let pred_end = pred.last_seen + self.interval_len;\n\n        if pred_end <= self.start_time {\n            return;\n        }\n\n        // If first range already covers (start_time, pred_end) return\n        if self\n            .interval_starts\n            .as_slice()\n            .first()\n            .filter(|v| **v == self.start_time)\n            .is_some()\n            && self\n                .interval_ends\n                .as_slice()\n                .first()\n                .filter(|v| **v >= pred_end)\n                .is_some()\n        {\n            return;\n        }\n\n        if self\n            .interval_starts\n            .as_slice()\n            .first()\n            .filter(|v| **v <= pred_end)\n            .is_some()\n        {\n            self.interval_starts.as_owned()[0] = self.start_time;\n        } else {\n            let start = self.start_time;\n            self.interval_starts.as_owned().insert(0, start);\n            self.interval_ends.as_owned().insert(0, pred_end);\n            self.num_intervals += 1;\n        }\n    }\n}\n\n#[pg_extern]\npub fn live_ranges(\n    agg: HeartbeatAgg<'static>,\n) -> TableIterator<'static, (name!(start, TimestampTz), name!(end, TimestampTz))> {\n    let starts = agg.interval_starts.clone();\n    let ends = agg.interval_ends.clone();\n    TableIterator::new(\n        starts\n            .into_iter()\n            .map(|x| x.into())\n            .zip(ends.into_iter().map(|x| x.into())),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_live_ranges(\n    sketch: HeartbeatAgg<'static>,\n    _accessor: AccessorLiveRanges,\n) -> TableIterator<'static, (name!(start, TimestampTz), name!(end, TimestampTz))> {\n    live_ranges(sketch)\n}\n\n#[pg_extern]\npub fn dead_ranges(\n    agg: HeartbeatAgg<'static>,\n) -> TableIterator<'static, (name!(start, TimestampTz), name!(end, TimestampTz))> {\n    if agg.num_intervals == 0 {\n        return TableIterator::new(std::iter::once((\n            agg.start_time.into(),\n            agg.end_time.into(),\n        )));\n    }\n\n    // Dead ranges are the opposite of the intervals stored in the aggregate\n    let mut starts = agg.interval_ends.clone().into_vec();\n    let mut ends = agg.interval_starts.clone().into_vec();\n\n    // Fix the first point depending on whether the aggregate starts in a live or dead range\n    if ends[0] == agg.start_time {\n        ends.remove(0);\n    } else {\n        starts.insert(0, agg.start_time);\n    }\n\n    // Fix the last point depending on whether the aggregate starts in a live or dead range\n    if *starts.last().unwrap() == agg.end_time {\n        starts.pop();\n    } else {\n        ends.push(agg.end_time);\n    }\n\n    TableIterator::new(\n        starts\n            .into_iter()\n            .map(|x| x.into())\n            .zip(ends.into_iter().map(|x| x.into())),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_dead_ranges(\n    sketch: HeartbeatAgg<'static>,\n    _accessor: AccessorDeadRanges,\n) -> TableIterator<'static, (name!(start, TimestampTz), name!(end, TimestampTz))> {\n    dead_ranges(sketch)\n}\n\n#[pg_extern]\npub fn uptime(agg: HeartbeatAgg<'static>) -> Interval {\n    agg.sum_live_intervals().into()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_uptime(\n    sketch: HeartbeatAgg<'static>,\n    _accessor: AccessorUptime,\n) -> Interval {\n    uptime(sketch)\n}\n\n#[pg_extern]\npub fn interpolated_uptime(\n    agg: HeartbeatAgg<'static>,\n    pred: Option<HeartbeatAgg<'static>>,\n) -> Interval {\n    uptime(interpolate_heartbeat_agg(agg, pred))\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_interpolated_uptime(\n    sketch: HeartbeatAgg<'static>,\n    accessor: HeartbeatInterpolatedUptimeAccessor<'static>,\n) -> Interval {\n    interpolated_uptime(sketch, accessor.pred())\n}\n\n#[pg_extern]\npub fn downtime(agg: HeartbeatAgg<'static>) -> Interval {\n    (agg.end_time - agg.start_time - agg.sum_live_intervals()).into()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_downtime(\n    sketch: HeartbeatAgg<'static>,\n    _accessor: AccessorDowntime,\n) -> Interval {\n    downtime(sketch)\n}\n\n#[pg_extern]\npub fn interpolated_downtime(\n    agg: HeartbeatAgg<'static>,\n    pred: Option<HeartbeatAgg<'static>>,\n) -> Interval {\n    downtime(interpolate_heartbeat_agg(agg, pred))\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_interpolated_downtime(\n    sketch: HeartbeatAgg<'static>,\n    accessor: HeartbeatInterpolatedDowntimeAccessor<'static>,\n) -> Interval {\n    interpolated_downtime(sketch, accessor.pred())\n}\n\n#[pg_extern]\npub fn live_at(agg: HeartbeatAgg<'static>, test: TimestampTz) -> bool {\n    let test = i64::from(test);\n\n    if test < agg.start_time || test > agg.end_time {\n        error!(\"unable to test for liveness outside of a heartbeat_agg's covered range\")\n    }\n\n    if agg.num_intervals == 0 {\n        return false;\n    }\n\n    let mut start_iter = agg.interval_starts.iter().enumerate().peekable();\n    while let Some((idx, val)) = start_iter.next() {\n        if test < val {\n            // Only possible if test shows up before first interval\n            return false;\n        }\n        if let Some((_, next_val)) = start_iter.peek() {\n            if test < *next_val {\n                return test < agg.interval_ends.as_slice()[idx];\n            }\n        }\n    }\n    // Fall out the loop if test > start of last interval\n    test < *agg.interval_ends.as_slice().last().unwrap()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_live_at(\n    sketch: HeartbeatAgg<'static>,\n    accessor: AccessorLiveAt,\n) -> bool {\n    let ts = TimestampTz(accessor.time.into());\n    live_at(sketch, ts)\n}\n\n#[pg_extern(name = \"interpolate\")]\nfn interpolate_heartbeat_agg(\n    agg: HeartbeatAgg<'static>,\n    pred: Option<HeartbeatAgg<'static>>,\n) -> HeartbeatAgg<'static> {\n    let mut r = agg.clone();\n    if let Some(pred) = pred {\n        r.interpolate_start(&pred);\n    }\n    r\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_interpolate(\n    sketch: HeartbeatAgg<'static>,\n    accessor: HeartbeatInterpolateAccessor<'static>,\n) -> HeartbeatAgg<'static> {\n    interpolate_heartbeat_agg(sketch, accessor.pred())\n}\n\n#[pg_extern]\npub fn num_live_ranges(agg: HeartbeatAgg<'static>) -> i64 {\n    agg.num_intervals as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_num_live_ranges(\n    agg: HeartbeatAgg<'static>,\n    _accessor: AccessorNumLiveRanges,\n) -> i64 {\n    num_live_ranges(agg)\n}\n\n#[pg_extern]\npub fn num_gaps(agg: HeartbeatAgg<'static>) -> i64 {\n    if agg.num_intervals == 0 {\n        return 1;\n    }\n    let mut count = agg.num_intervals - 1;\n    if agg.interval_starts.slice()[0] != agg.start_time {\n        count += 1;\n    }\n    if agg.interval_ends.slice()[agg.num_intervals as usize - 1] != agg.end_time {\n        count += 1;\n    }\n    count as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_num_gaps(agg: HeartbeatAgg<'static>, _accessor: AccessorNumGaps) -> i64 {\n    num_gaps(agg)\n}\n\n#[pg_extern]\npub fn trim_to(\n    agg: HeartbeatAgg<'static>,\n    start: default!(Option<crate::raw::TimestampTz>, \"NULL\"),\n    duration: default!(Option<crate::raw::Interval>, \"NULL\"),\n) -> HeartbeatAgg<'static> {\n    if let Some(start) = start {\n        let end = duration.map(|intv| crate::datum_utils::ts_interval_sum_to_ms(&start, &intv));\n        agg.trim_to(Some(i64::from(start)), end)\n    } else {\n        let end = duration.map(|intv| {\n            crate::datum_utils::ts_interval_sum_to_ms(&TimestampTz::from(agg.start_time), &intv)\n        });\n        agg.trim_to(None, end)\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_heartbeat_agg_trim_to(\n    agg: HeartbeatAgg<'static>,\n    accessor: HeartbeatTrimToAccessor,\n) -> HeartbeatAgg<'static> {\n    let end = if accessor.end == 0 {\n        None\n    } else {\n        Some(accessor.end)\n    };\n    agg.trim_to(Some(accessor.start), end)\n}\n\nimpl From<HeartbeatAgg<'static>> for HeartbeatTransState {\n    fn from(agg: HeartbeatAgg<'static>) -> Self {\n        HeartbeatTransState {\n            start: agg.start_time,\n            end: agg.end_time,\n            last: agg.last_seen,\n            interval_len: agg.interval_len,\n            buffer: vec![],\n            liveness: agg\n                .interval_starts\n                .iter()\n                .zip(agg.interval_ends.iter())\n                .collect(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn heartbeat_trans(\n    state: Internal,\n    heartbeat: TimestampTz,\n    start: TimestampTz,\n    length: Interval,\n    liveness_duration: Interval,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    heartbeat_trans_inner(\n        unsafe { state.to_inner() },\n        heartbeat,\n        start,\n        length,\n        liveness_duration,\n        fcinfo,\n    )\n    .internal()\n}\npub fn heartbeat_trans_inner(\n    state: Option<Inner<HeartbeatTransState>>,\n    heartbeat: TimestampTz,\n    start: TimestampTz,\n    length: Interval,\n    liveness_duration: Interval,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<HeartbeatTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = state.unwrap_or_else(|| {\n                let length = interval_to_ms(&start, &length);\n                let interval = interval_to_ms(&start, &liveness_duration);\n                let start = start.into();\n                HeartbeatTransState::new(start, start + length, interval).into()\n            });\n            state.insert(heartbeat.into());\n            Some(state)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn heartbeat_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<HeartbeatAgg<'static>> {\n    heartbeat_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\npub fn heartbeat_final_inner(\n    state: Option<Inner<HeartbeatTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<HeartbeatAgg<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            state.map(|mut s| {\n                s.process_batch();\n                let (starts, mut ends): (Vec<i64>, Vec<i64>) =\n                    s.liveness.clone().into_iter().unzip();\n\n                // Trim last interval to end of aggregate's range\n                if let Some(last) = ends.last_mut() {\n                    if *last > s.end {\n                        *last = s.end;\n                    }\n                }\n\n                flatten!(HeartbeatAgg {\n                    start_time: s.start,\n                    end_time: s.end,\n                    last_seen: s.last,\n                    interval_len: s.interval_len,\n                    num_intervals: starts.len() as u64,\n                    interval_starts: starts.into(),\n                    interval_ends: ends.into(),\n                })\n            })\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn heartbeat_rollup_trans(\n    state: Internal,\n    value: Option<HeartbeatAgg<'static>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    heartbeat_rollup_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn heartbeat_rollup_trans_inner(\n    state: Option<Inner<HeartbeatTransState>>,\n    value: Option<HeartbeatAgg<'static>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<HeartbeatTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, value) {\n            (a, None) => a,\n            (None, Some(a)) => Some(HeartbeatTransState::from(a).into()),\n            (Some(mut a), Some(b)) => {\n                a.combine(b.into());\n                Some(a)\n            }\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE heartbeat_agg(\\n\\\n        heartbeat TIMESTAMPTZ, agg_start TIMESTAMPTZ, agg_duration INTERVAL, heartbeat_liveness INTERVAL\\n\\\n    ) (\\n\\\n        sfunc = heartbeat_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = heartbeat_final\\n\\\n    );\\n\\\n\",\n    name = \"heartbeat_agg\",\n    requires = [\n        heartbeat_trans,\n        heartbeat_final,\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        HeartbeatAgg\\n\\\n    ) (\\n\\\n        sfunc = heartbeat_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = heartbeat_final\\n\\\n    );\\n\\\n\",\n    name = \"heartbeat_agg_rollup\",\n    requires = [heartbeat_rollup_trans, heartbeat_final,],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n\n    #[pg_test]\n    pub fn test_heartbeat_trans_state() {\n        let mut state = HeartbeatTransState::new(0, 500, 10);\n        state.insert(100);\n        state.insert(200);\n        state.insert(250);\n        state.insert(220);\n        state.insert(210);\n        state.insert(300);\n\n        assert_eq!(state.get_buffer().len(), 6);\n\n        state.process_batch();\n        assert_eq!(state.get_buffer().len(), 0);\n\n        let mut it = state.get_liveness().iter();\n        assert_eq!(*it.next().unwrap(), (100, 110));\n        assert_eq!(*it.next().unwrap(), (200, 230));\n        assert_eq!(*it.next().unwrap(), (250, 260));\n        assert_eq!(*it.next().unwrap(), (300, 310));\n        assert!(it.next().is_none());\n\n        state.insert(400);\n        state.insert(350);\n        state.process_batch();\n\n        let mut it = state.get_liveness().iter();\n        assert_eq!(*it.next().unwrap(), (100, 110));\n        assert_eq!(*it.next().unwrap(), (200, 230));\n        assert_eq!(*it.next().unwrap(), (250, 260));\n        assert_eq!(*it.next().unwrap(), (300, 310));\n        assert_eq!(*it.next().unwrap(), (350, 360));\n        assert_eq!(*it.next().unwrap(), (400, 410));\n        assert!(it.next().is_none());\n\n        state.insert(80);\n        state.insert(190);\n        state.insert(210);\n        state.insert(230);\n        state.insert(240);\n        state.insert(310);\n        state.insert(395);\n        state.insert(408);\n        state.process_batch();\n\n        let mut it = state.get_liveness().iter();\n        assert_eq!(*it.next().unwrap(), (80, 90));\n        assert_eq!(*it.next().unwrap(), (100, 110));\n        assert_eq!(*it.next().unwrap(), (190, 260));\n        assert_eq!(*it.next().unwrap(), (300, 320));\n        assert_eq!(*it.next().unwrap(), (350, 360));\n        assert_eq!(*it.next().unwrap(), (395, 418));\n        assert!(it.next().is_none());\n    }\n\n    #[pg_test]\n    pub fn test_heartbeat_agg() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\"CREATE TABLE liveness(heartbeat TIMESTAMPTZ)\", None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO liveness VALUES\n                ('01-01-2020 0:2:20 UTC'),\n                ('01-01-2020 0:10 UTC'),\n                ('01-01-2020 0:17 UTC'),\n                ('01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:35 UTC'),\n                ('01-01-2020 0:40 UTC'),\n                ('01-01-2020 0:35 UTC'),\n                ('01-01-2020 0:40 UTC'),\n                ('01-01-2020 0:40 UTC'),\n                ('01-01-2020 0:50:30 UTC'),\n                ('01-01-2020 1:00 UTC'),\n                ('01-01-2020 1:08 UTC'),\n                ('01-01-2020 1:18 UTC'),\n                ('01-01-2020 1:28 UTC'),\n                ('01-01-2020 1:38:01 UTC'),\n                ('01-01-2020 1:40 UTC'),\n                ('01-01-2020 1:40:01 UTC'),\n                ('01-01-2020 1:50:01 UTC'),\n                ('01-01-2020 1:57 UTC'),\n                ('01-01-2020 1:59:50 UTC')\n            \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut result = client.update(\n                \"SELECT live_ranges(heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m'))::TEXT\n                FROM liveness\", None, &[]).unwrap();\n\n            let mut arrow_result = client.update(\n                \"SELECT (heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') -> live_ranges())::TEXT\n                FROM liveness\", None, &[]).unwrap();\n\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 00:02:20+00\\\",\\\"2020-01-01 00:27:00+00\\\")\"\n            );\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 00:30:00+00\\\",\\\"2020-01-01 00:50:00+00\\\")\"\n            );\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 00:50:30+00\\\",\\\"2020-01-01 01:38:00+00\\\")\"\n            );\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 01:38:01+00\\\",\\\"2020-01-01 02:00:00+00\\\")\"\n            );\n            assert!(result.next().is_none());\n            assert!(arrow_result.next().is_none());\n\n            let mut result = client.update(\n                \"SELECT dead_ranges(heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m'))::TEXT\n                FROM liveness\", None, &[]).unwrap();\n\n            let mut arrow_result = client.update(\n                \"SELECT (heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') -> dead_ranges())::TEXT\n                FROM liveness\", None, &[]).unwrap();\n\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-01 00:02:20+00\\\")\"\n            );\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 00:27:00+00\\\",\\\"2020-01-01 00:30:00+00\\\")\"\n            );\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 00:50:00+00\\\",\\\"2020-01-01 00:50:30+00\\\")\"\n            );\n            let test = arrow_result.next().unwrap()[1]\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                test\n            );\n            assert_eq!(\n                test,\n                \"(\\\"2020-01-01 01:38:00+00\\\",\\\"2020-01-01 01:38:01+00\\\")\"\n            );\n            assert!(result.next().is_none());\n            assert!(arrow_result.next().is_none());\n\n            let result = client\n                .update(\n                    \"SELECT uptime(heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m'))::TEXT\n                FROM liveness\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\"01:54:09\", result);\n\n            let result = client.update(\n                \"SELECT (heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') -> uptime())::TEXT\n                FROM liveness\", None, &[]).unwrap().first().get_one::<String>().unwrap().unwrap();\n            assert_eq!(\"01:54:09\", result);\n\n            let result = client\n                .update(\n                    \"SELECT downtime(heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m'))::TEXT\n                FROM liveness\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\"00:05:51\", result);\n\n            let result = client.update(\n                \"SELECT (heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') -> downtime())::TEXT\n                FROM liveness\", None, &[]).unwrap().first().get_one::<String>().unwrap().unwrap();\n            assert_eq!(\"00:05:51\", result);\n\n            let (result1, result2, result3) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT live_at(agg, '01-01-2020 00:01:00 UTC')::TEXT, \n                    live_at(agg, '01-01-2020 00:05:00 UTC')::TEXT,\n                    live_at(agg, '01-01-2020 00:30:00 UTC')::TEXT FROM agg\", None, &[])\n                .unwrap().first()\n                .get_three::<String, String, String>().unwrap();\n\n            let result4 =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT live_at(agg, '01-01-2020 01:38:00 UTC')::TEXT FROM agg\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n\n            assert_eq!(result1.unwrap(), \"false\"); // outside ranges\n            assert_eq!(result2.unwrap(), \"true\"); // inside ranges\n            assert_eq!(result3.unwrap(), \"true\"); // first point of range\n            assert_eq!(result4.unwrap(), \"false\"); // last point of range\n\n            let (result1, result2, result3) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT (agg -> live_at('01-01-2020 00:01:00 UTC'))::TEXT, \n                    (agg -> live_at('01-01-2020 00:05:00 UTC'))::TEXT,\n                    (agg -> live_at('01-01-2020 00:30:00 UTC'))::TEXT FROM agg\", None, &[])\n                .unwrap().first()\n                .get_three::<String, String, String>().unwrap();\n\n            let result4 =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT (agg -> live_at('01-01-2020 01:38:00 UTC'))::TEXT FROM agg\", None, &[])\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n\n            assert_eq!(result1.unwrap(), \"false\"); // outside ranges\n            assert_eq!(result2.unwrap(), \"true\"); // inside ranges\n            assert_eq!(result3.unwrap(), \"true\"); // first point of range\n            assert_eq!(result4.unwrap(), \"false\"); // last point of range\n\n            let (result1, result2) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT num_live_ranges(agg), num_gaps(agg) FROM agg\", None, &[])\n                .unwrap().first()\n                .get_two::<i64, i64>().unwrap();\n\n            assert_eq!(result1.unwrap(), 4);\n            assert_eq!(result2.unwrap(), 4);\n\n            let (result1, result2) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT agg->num_live_ranges(), agg->num_gaps() FROM agg\", None, &[])\n                .unwrap().first()\n                .get_two::<i64, i64>().unwrap();\n\n            assert_eq!(result1.unwrap(), 4);\n            assert_eq!(result2.unwrap(), 4);\n        })\n    }\n\n    #[pg_test]\n    pub fn test_heartbeat_rollup() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE heartbeats(time timestamptz, batch timestamptz)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client.update(\n                \"INSERT INTO heartbeats VALUES\n                    ('01-01-2020 3:02:20 UTC'::timestamptz, '01-01-2020 3:00:00 UTC'::timestamptz),\n                    ('01-01-2020 3:03:10 UTC'::timestamptz, '01-01-2020 3:00:00 UTC'::timestamptz),\n                    ('01-01-2020 3:04:07 UTC'::timestamptz, '01-01-2020 3:00:00 UTC'::timestamptz),\n                    ('01-01-2020 7:19:20 UTC'::timestamptz, '01-01-2020 7:00:00 UTC'::timestamptz),\n                    ('01-01-2020 7:39:20 UTC'::timestamptz, '01-01-2020 7:00:00 UTC'::timestamptz),\n                    ('01-01-2020 7:59:20 UTC'::timestamptz, '01-01-2020 7:00:00 UTC'::timestamptz),\n                    ('01-01-2020 8:00:10 UTC'::timestamptz, '01-01-2020 8:00:00 UTC'::timestamptz),\n                    ('01-01-2020 8:59:10 UTC'::timestamptz, '01-01-2020 8:00:00 UTC'::timestamptz),\n                    ('01-01-2020 23:34:20 UTC'::timestamptz, '01-01-2020 23:00:00 UTC'::timestamptz),\n                    ('01-01-2020 23:37:20 UTC'::timestamptz, '01-01-2020 23:00:00 UTC'::timestamptz),\n                    ('01-01-2020 23:38:05 UTC'::timestamptz, '01-01-2020 23:00:00 UTC'::timestamptz),\n                    ('01-01-2020 23:39:00 UTC'::timestamptz, '01-01-2020 23:00:00 UTC'::timestamptz)\",\n                None,\n                &[],\n            ).unwrap();\n\n            let result = client\n                .update(\n                    \"WITH aggs AS (\n                    SELECT heartbeat_agg(time, batch, '1h', '1m')\n                    FROM heartbeats \n                    GROUP BY batch\n                ) SELECT rollup(heartbeat_agg)::TEXT FROM aggs\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\"(version:1,start_time:631162800000000,end_time:631238400000000,last_seen:631237140000000,interval_len:60000000,num_intervals:7,interval_starts:[631162940000000,631178360000000,631179560000000,631180760000000,631184350000000,631236860000000,631237040000000],interval_ends:[631163107000000,631178420000000,631179620000000,631180870000000,631184410000000,631236920000000,631237200000000])\", result);\n        })\n    }\n\n    #[pg_test]\n    pub fn test_heartbeat_combining_rollup() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\"CREATE TABLE aggs(agg heartbeatagg)\", None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO aggs SELECT heartbeat_agg(hb, '01-01-2020 UTC', '1h', '10m')\n                FROM (VALUES\n                    ('01-01-2020 0:2:20 UTC'::timestamptz),\n                    ('01-01-2020 0:10 UTC'::timestamptz),\n                    ('01-01-2020 0:17 UTC'::timestamptz),\n                    ('01-01-2020 0:30 UTC'::timestamptz),\n                    ('01-01-2020 0:35 UTC'::timestamptz),\n                    ('01-01-2020 0:40 UTC'::timestamptz),\n                    ('01-01-2020 0:50:30 UTC'::timestamptz)\n                ) AS _(hb)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO aggs SELECT heartbeat_agg(hb, '01-01-2020 0:30 UTC', '1h', '10m')\n                    FROM (VALUES\n                    ('01-01-2020 0:35 UTC'::timestamptz),\n                    ('01-01-2020 0:40 UTC'::timestamptz),\n                    ('01-01-2020 0:40 UTC'::timestamptz),\n                    ('01-01-2020 1:08 UTC'::timestamptz),\n                    ('01-01-2020 1:18 UTC'::timestamptz)\n                ) AS _(hb)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO aggs SELECT heartbeat_agg(hb, '01-01-2020 1:00 UTC', '1h', '10m')\n                FROM (VALUES\n                    ('01-01-2020 1:00 UTC'::timestamptz),\n                    ('01-01-2020 1:28 UTC'::timestamptz),\n                    ('01-01-2020 1:38:01 UTC'::timestamptz),\n                    ('01-01-2020 1:40 UTC'::timestamptz),\n                    ('01-01-2020 1:40:01 UTC'::timestamptz),\n                    ('01-01-2020 1:50:01 UTC'::timestamptz),\n                    ('01-01-2020 1:57 UTC'::timestamptz),\n                    ('01-01-2020 1:59:50 UTC'::timestamptz)\n                ) AS _(hb)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut result = client\n                .update(\n                    \"SELECT dead_ranges(rollup(agg))::TEXT\n                FROM aggs\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-01 00:02:20+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:27:00+00\\\",\\\"2020-01-01 00:30:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:50:00+00\\\",\\\"2020-01-01 00:50:30+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:38:00+00\\\",\\\"2020-01-01 01:38:01+00\\\")\"\n            );\n            assert!(result.next().is_none());\n        });\n    }\n\n    #[pg_test]\n    pub fn test_heartbeat_trim_to() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\"CREATE TABLE liveness(heartbeat TIMESTAMPTZ)\", None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO liveness VALUES\n                ('01-01-2020 0:2:20 UTC'),\n                ('01-01-2020 0:10 UTC'),\n                ('01-01-2020 0:17 UTC'),\n                ('01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:35 UTC'),\n                ('01-01-2020 0:40 UTC'),\n                ('01-01-2020 0:35 UTC'),\n                ('01-01-2020 0:40 UTC'),\n                ('01-01-2020 0:40 UTC'),\n                ('01-01-2020 0:50:30 UTC'),\n                ('01-01-2020 1:00 UTC'),\n                ('01-01-2020 1:08 UTC'),\n                ('01-01-2020 1:18 UTC'),\n                ('01-01-2020 1:28 UTC'),\n                ('01-01-2020 1:38:01 UTC'),\n                ('01-01-2020 1:40 UTC'),\n                ('01-01-2020 1:40:01 UTC'),\n                ('01-01-2020 1:50:01 UTC'),\n                ('01-01-2020 1:57 UTC'),\n                ('01-01-2020 1:59:50 UTC')\n            \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (result1, result2, result3) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness),\n                    trimmed AS (SELECT trim_to(agg, '01-01-2020 0:30 UTC', '1h') AS agg FROM agg)\n                    SELECT uptime(agg)::TEXT, num_gaps(agg), live_at(agg, '01-01-2020 0:50:25 UTC')::TEXT FROM trimmed\", None, &[])\n                .unwrap().first()\n                .get_three::<String, i64, String>().unwrap();\n\n            assert_eq!(result1.unwrap(), \"00:59:30\");\n            assert_eq!(result2.unwrap(), 1);\n            assert_eq!(result3.unwrap(), \"false\");\n\n            let (result1, result2, result3) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness),\n                    trimmed AS (SELECT trim_to(agg, duration=>'30m') AS agg FROM agg)\n                    SELECT uptime(agg)::TEXT, num_gaps(agg), live_at(agg, '01-01-2020 0:20:25 UTC')::TEXT FROM trimmed\", None, &[])\n                .unwrap().first()\n                .get_three::<String, i64, String>().unwrap();\n\n            assert_eq!(result1.unwrap(), \"00:24:40\");\n            assert_eq!(result2.unwrap(), 2);\n            assert_eq!(result3.unwrap(), \"true\");\n\n            let (result1, result2, result3) =\n                client.update(\n                    \"WITH agg AS (SELECT heartbeat_agg(heartbeat, '01-01-2020 UTC', '2h', '10m') AS agg FROM liveness)\n                    SELECT agg -> trim_to('01-01-2020 1:40:00 UTC'::timestamptz) -> num_gaps(),\n                    (agg -> trim_to('01-01-2020 00:50:00 UTC'::timestamptz, '30s') -> uptime())::TEXT,\n                    agg -> trim_to('01-01-2020 00:28:00 UTC'::timestamptz, '22m15s') -> num_live_ranges() FROM agg\", None, &[])\n                .unwrap().first()\n                .get_three::<i64, String, i64>().unwrap();\n\n            assert_eq!(result1.unwrap(), 0);\n            assert_eq!(result2.unwrap(), \"00:00:00\");\n            assert_eq!(result3.unwrap(), 1);\n        });\n    }\n\n    #[pg_test]\n    pub fn test_heartbeat_agg_interpolation() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE liveness(heartbeat TIMESTAMPTZ, start TIMESTAMPTZ)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO liveness VALUES\n                ('01-01-2020 0:2:20 UTC', '01-01-2020 0:0 UTC'),\n                ('01-01-2020 0:10 UTC', '01-01-2020 0:0 UTC'),\n                ('01-01-2020 0:17 UTC', '01-01-2020 0:0 UTC'),\n                ('01-01-2020 0:30 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:35 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:40 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:35 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:40 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:40 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 0:50:30 UTC', '01-01-2020 0:30 UTC'),\n                ('01-01-2020 1:00:30 UTC', '01-01-2020 1:00 UTC'),\n                ('01-01-2020 1:08 UTC', '01-01-2020 1:00 UTC'),\n                ('01-01-2020 1:18 UTC', '01-01-2020 1:00 UTC'),\n                ('01-01-2020 1:28 UTC', '01-01-2020 1:00 UTC'),\n                ('01-01-2020 1:38:01 UTC', '01-01-2020 1:30 UTC'),\n                ('01-01-2020 1:40 UTC', '01-01-2020 1:30 UTC'),\n                ('01-01-2020 1:40:01 UTC', '01-01-2020 1:30 UTC'),\n                ('01-01-2020 1:50:01 UTC', '01-01-2020 1:30 UTC'),\n                ('01-01-2020 1:57 UTC', '01-01-2020 1:30 UTC'),\n                ('01-01-2020 1:59:50 UTC', '01-01-2020 1:30 UTC')\n            \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start),\n                t AS (\n                    SELECT start,\n                        interpolate(agg, LAG (agg) OVER (ORDER BY start)) AS agg \n                    FROM s)\n                SELECT downtime(agg)::TEXT FROM t;\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:05:20\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:30\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:00\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:01\"\n            );\n            assert!(result.next().is_none());\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start),\n                t AS (\n                    SELECT start,\n                        interpolate(agg, LAG (agg) OVER (ORDER BY start)) AS agg \n                    FROM s)\n                SELECT live_ranges(agg)::TEXT FROM t;\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:02:20+00\\\",\\\"2020-01-01 00:27:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:30:00+00\\\",\\\"2020-01-01 00:50:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:50:30+00\\\",\\\"2020-01-01 01:00:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:00:00+00\\\",\\\"2020-01-01 01:30:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:30:00+00\\\",\\\"2020-01-01 01:38:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:38:01+00\\\",\\\"2020-01-01 02:00:00+00\\\")\"\n            );\n            assert!(result.next().is_none());\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start),\n                t AS (\n                    SELECT start,\n                        agg -> interpolate(LAG (agg) OVER (ORDER BY start)) AS agg \n                    FROM s)\n                SELECT live_ranges(agg)::TEXT FROM t;\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:02:20+00\\\",\\\"2020-01-01 00:27:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:30:00+00\\\",\\\"2020-01-01 00:50:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 00:50:30+00\\\",\\\"2020-01-01 01:00:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:00:00+00\\\",\\\"2020-01-01 01:30:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:30:00+00\\\",\\\"2020-01-01 01:38:00+00\\\")\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"(\\\"2020-01-01 01:38:01+00\\\",\\\"2020-01-01 02:00:00+00\\\")\"\n            );\n            assert!(result.next().is_none());\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start)\n                SELECT interpolated_uptime(agg, LAG (agg) OVER (ORDER BY start))::TEXT\n                FROM s\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:24:40\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:29:30\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:30:00\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:29:59\"\n            );\n            assert!(result.next().is_none());\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start)\n                SELECT (agg -> interpolated_uptime(LAG (agg) OVER (ORDER BY start)))::TEXT\n                FROM s\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:24:40\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:29:30\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:30:00\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:29:59\"\n            );\n            assert!(result.next().is_none());\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start)\n                SELECT interpolated_downtime(agg, LAG (agg) OVER (ORDER BY start))::TEXT\n                FROM s\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:05:20\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:30\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:00\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:01\"\n            );\n            assert!(result.next().is_none());\n\n            let mut result = client\n                .update(\n                    \"WITH s AS (\n                    SELECT start,\n                        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg \n                    FROM liveness \n                    GROUP BY start)\n                SELECT (agg -> interpolated_downtime(LAG (agg) OVER (ORDER BY start)))::TEXT\n                FROM s\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:05:20\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:30\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:00\"\n            );\n            assert_eq!(\n                result.next().unwrap()[1]\n                    .value::<String>()\n                    .unwrap()\n                    .unwrap(),\n                \"00:00:01\"\n            );\n            assert!(result.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    fn test_heartbeat_agg_text_io() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\"CREATE TABLE liveness(heartbeat TIMESTAMPTZ)\", None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO liveness VALUES\n                ('01-01-2020 0:2:20 UTC'),\n                ('01-01-2020 0:10 UTC'),\n                ('01-01-2020 0:17 UTC')\n            \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"SELECT heartbeat_agg(heartbeat, '01-01-2020', '30m', '5m')::TEXT\n                    FROM liveness;\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(version:1,start_time:631152000000000,end_time:631153800000000,last_seen:631153020000000,interval_len:300000000,num_intervals:3,interval_starts:[631152140000000,631152600000000,631153020000000],interval_ends:[631152440000000,631152900000000,631153320000000])\";\n\n            assert_eq!(output, Some(expected.into()));\n\n            let estimate = client\n                .update(\n                    &format!(\"SELECT uptime('{expected}'::heartbeatagg)::TEXT\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(estimate.unwrap().as_str(), \"00:15:00\");\n        });\n    }\n\n    #[pg_test]\n    fn test_heartbeat_agg_byte_io() {\n        use std::ptr;\n\n        // Create a heartbeat agg from 0 to 250 with intervals from 40-50, 60-85, and 100-110\n        let state = heartbeat_trans_inner(\n            None,\n            40.into(),\n            0.into(),\n            250.into(),\n            10.into(),\n            ptr::null_mut(),\n        );\n        let state = heartbeat_trans_inner(\n            state,\n            60.into(),\n            0.into(),\n            250.into(),\n            10.into(),\n            ptr::null_mut(),\n        );\n        let state = heartbeat_trans_inner(\n            state,\n            65.into(),\n            0.into(),\n            250.into(),\n            10.into(),\n            ptr::null_mut(),\n        );\n        let state = heartbeat_trans_inner(\n            state,\n            75.into(),\n            0.into(),\n            250.into(),\n            10.into(),\n            ptr::null_mut(),\n        );\n        let state = heartbeat_trans_inner(\n            state,\n            100.into(),\n            0.into(),\n            250.into(),\n            10.into(),\n            ptr::null_mut(),\n        );\n\n        let agg = heartbeat_final_inner(state, ptr::null_mut())\n            .expect(\"failed to build finalized heartbeat_agg\");\n        let serial = agg.to_pg_bytes();\n\n        let expected = [\n            128, 1, 0, 0, // header\n            1, // version\n            0, 0, 0, // padding\n            0, 0, 0, 0, 0, 0, 0, 0, // start_time\n            250, 0, 0, 0, 0, 0, 0, 0, // end_time\n            100, 0, 0, 0, 0, 0, 0, 0, // last_seen\n            10, 0, 0, 0, 0, 0, 0, 0, // interval_len\n            3, 0, 0, 0, 0, 0, 0, 0, // num_intervals\n            40, 0, 0, 0, 0, 0, 0, 0, // interval_starts[0]\n            60, 0, 0, 0, 0, 0, 0, 0, // interval_starts[1]\n            100, 0, 0, 0, 0, 0, 0, 0, // interval_starts[2]\n            50, 0, 0, 0, 0, 0, 0, 0, // interval_ends[0]\n            85, 0, 0, 0, 0, 0, 0, 0, // interval_ends[1]\n            110, 0, 0, 0, 0, 0, 0, 0, // interval_ends[2]\n        ];\n        assert_eq!(serial, expected);\n    }\n\n    #[pg_test]\n    fn test_rollup_overlap() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET TIMEZONE to UTC\", None, &[]).unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE poc(ts TIMESTAMPTZ, batch TIMESTAMPTZ)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"INSERT INTO poc VALUES\n                    ('1-1-2020 0:50 UTC', '1-1-2020 0:00 UTC'),\n                    ('1-1-2020 1:10 UTC', '1-1-2020 0:00 UTC'),\n                    ('1-1-2020 1:00 UTC', '1-1-2020 1:00 UTC')\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"WITH rollups AS (\n                        SELECT heartbeat_agg(ts, batch, '2h', '20m') \n                        FROM poc \n                        GROUP BY batch \n                        ORDER BY batch\n                    )\n                    SELECT live_ranges(rollup(heartbeat_agg))::TEXT \n                    FROM rollups\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(\\\"2020-01-01 00:50:00+00\\\",\\\"2020-01-01 01:30:00+00\\\")\";\n\n            assert_eq!(output, Some(expected.into()));\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/hyperloglog.rs",
    "content": "#![allow(clippy::identity_op)] // clippy gets confused by flat_serialize! enums\n\nuse std::{\n    convert::TryInto,\n    hash::{Hash, Hasher},\n};\n\nuse serde::{Deserialize, Serialize};\n\nuse pg_sys::{Datum, Oid};\nuse pgrx::*;\n\nuse crate::{\n    accessors::{AccessorDistinctCount, AccessorStderror},\n    aggregate_utils::{get_collation, in_aggregate_context},\n    datum_utils::DatumHashBuilder,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n    serialization::{PgCollationId, ShortTypeId},\n};\n\nuse hyperloglogplusplus::{HyperLogLog as HLL, HyperLogLogStorage};\n\n// pgrx doesn't implement Eq/Hash but it's okay here since we treat Datums as raw bytes\n#[derive(Debug, Copy, Clone, PartialEq)]\nstruct HashableDatum(Datum);\nimpl Eq for HashableDatum {}\n#[allow(clippy::derived_hash_with_manual_eq)] // partialeq and hash implementations match\nimpl Hash for HashableDatum {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        self.0.value().hash(state)\n    }\n}\n\n#[derive(Serialize, Deserialize, Clone, PartialEq, Eq)]\npub struct HyperLogLogTrans {\n    logger: HLL<'static, HashableDatum, DatumHashBuilder>,\n}\n\nuse crate::raw::AnyElement;\n\n#[pg_extern(immutable, parallel_safe)]\npub fn hyperloglog_trans(\n    state: Internal,\n    size: i32,\n    // TODO we want to use crate::raw::AnyElement but it doesn't work for some reason...\n    value: Option<AnyElement>,\n    fc: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    // let state: Internal = Internal::from_polymorphic_datum();\n    hyperloglog_trans_inner(unsafe { state.to_inner() }, size, value, fc, unsafe {\n        pgrx::pg_getarg_type(fc, 2)\n    })\n    .internal()\n}\n\nconst APPROX_COUNT_DISTINCT_DEFAULT_SIZE: i32 = 32768;\n\n/// Similar to hyperloglog_trans(), except size is set to a default of 32,768\n#[pg_extern(immutable, parallel_safe)]\npub fn approx_count_distinct_trans(\n    state: Internal,\n    // TODO we want to use crate::raw::AnyElement but it doesn't work for some reason...\n    value: Option<AnyElement>,\n    fc: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    // let state: Internal = Internal::from_polymorphic_datum();\n    hyperloglog_trans_inner(\n        unsafe { state.to_inner() },\n        APPROX_COUNT_DISTINCT_DEFAULT_SIZE,\n        value,\n        fc,\n        unsafe { pgrx::pg_getarg_type(fc, 1) },\n    )\n    .internal()\n}\n\npub fn hyperloglog_trans_inner(\n    state: Option<Inner<HyperLogLogTrans>>,\n    size: i32,\n    value: Option<AnyElement>,\n    fc: pg_sys::FunctionCallInfo,\n    arg_type: pg_sys::Oid,\n) -> Option<Inner<HyperLogLogTrans>> {\n    unsafe {\n        in_aggregate_context(fc, || {\n            //TODO is this the right way to handle NULL?\n            let value = match value {\n                None => return state,\n                Some(value) => value.0,\n            };\n            let mut state = match state {\n                None => {\n                    // TODO specialize hash function for bytea types?\n                    //      ints? floats? uuids? other primitive types?\n                    let size: usize = size.try_into().unwrap();\n                    let b = size.checked_next_power_of_two().unwrap().trailing_zeros();\n\n                    if !(4..=18).contains(&b) {\n                        error!(\n                            \"Invalid value for size {}. \\\n                            Size must be between 16 and 262144, \\\n                            though less than 1024 not recommended\",\n                            size\n                        )\n                    }\n\n                    let typ = arg_type;\n                    let collation = get_collation(fc);\n                    let hasher = DatumHashBuilder::from_type_id(typ, collation);\n                    let trans = HyperLogLogTrans {\n                        logger: HLL::new(b as u8, hasher),\n                    };\n                    trans.into()\n                }\n                Some(state) => state,\n            };\n            state.logger.add(&HashableDatum(value));\n            Some(state)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn hyperloglog_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { hyperloglog_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\npub fn hyperloglog_combine_inner(\n    state1: Option<Inner<HyperLogLogTrans>>,\n    state2: Option<Inner<HyperLogLogTrans>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<HyperLogLogTrans>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(state2)) => Some(state2.clone().into()),\n            (Some(state1), None) => Some(state1.clone().into()),\n            (Some(state1), Some(state2)) => {\n                let mut logger = state1.logger.clone();\n                logger.merge_in(&state2.logger);\n                Some(HyperLogLogTrans { logger }.into())\n            }\n        })\n    }\n}\n\nuse crate::raw::bytea;\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn hyperloglog_serialize(state: Internal) -> bytea {\n    let mut state = state;\n    let state: &mut HyperLogLogTrans = unsafe { state.get_mut().unwrap() };\n    state.logger.merge_all();\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn hyperloglog_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    hyperloglog_deserialize_inner(bytes).internal()\n}\npub fn hyperloglog_deserialize_inner(bytes: bytea) -> Inner<HyperLogLogTrans> {\n    let i: HyperLogLogTrans = crate::do_deserialize!(bytes, HyperLogLogTrans);\n    i.into()\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct HyperLogLog<'input> {\n        #[flat_serialize::flatten]\n        log: Storage<'input>,\n    }\n}\n\nflat_serialize_macro::flat_serialize! {\n    #[derive(Debug, Serialize, Deserialize)]\n    enum Storage<'a> {\n        storage_kind: u64,\n        Sparse: 1 {\n            num_compressed: u64,\n            // Oids are stored in postgres arrays, so it should be safe to store them\n            // in our types as long as we do send/recv and in/out correctly\n            // see https://github.com/postgres/postgres/blob/b8d0cda53377515ac61357ec4a60e85ca873f486/src/include/utils/array.h#L90\n            element_type: ShortTypeId,\n            collation: PgCollationId,\n            compressed_bytes: u32,\n            precision: u8,\n            compressed: [u8; self.compressed_bytes],\n        },\n        Dense: 2 {\n            // Oids are stored in postgres arrays, so it should be safe to store them\n            // in our types as long as we do send/recv and in/out correctly\n            // see https://github.com/postgres/postgres/blob/b8d0cda53377515ac61357ec4a60e85ca873f486/src/include/utils/array.h#L90\n            element_type: ShortTypeId,\n            collation: PgCollationId,\n            precision: u8,\n            registers: [u8; 1 + (1usize << self.precision) * 6 / 8] //TODO should we just store len?\n        },\n    }\n}\n\nron_inout_funcs!(HyperLogLog<'input>);\n\n#[pg_extern(immutable, parallel_safe)]\nfn hyperloglog_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<HyperLogLog<'static>> {\n    hyperloglog_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\nfn hyperloglog_final_inner(\n    state: Option<Inner<HyperLogLogTrans>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<HyperLogLog<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state,\n            };\n\n            flatten_log(&mut state.logger).into()\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE hyperloglog(size integer, value AnyElement)\\n\\\n    (\\n\\\n        stype = internal,\\n\\\n        sfunc = hyperloglog_trans,\\n\\\n        finalfunc = hyperloglog_final,\\n\\\n        combinefunc = hyperloglog_combine,\\n\\\n        serialfunc = hyperloglog_serialize,\\n\\\n        deserialfunc = hyperloglog_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"hll_agg\",\n    requires = [\n        hyperloglog_trans,\n        hyperloglog_final,\n        hyperloglog_combine,\n        hyperloglog_serialize,\n        hyperloglog_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE approx_count_distinct(value AnyElement)\\n\\\n    (\\n\\\n        stype = internal,\\n\\\n        sfunc = approx_count_distinct_trans,\\n\\\n        finalfunc = hyperloglog_final,\\n\\\n        combinefunc = hyperloglog_combine,\\n\\\n        serialfunc = hyperloglog_serialize,\\n\\\n        deserialfunc = hyperloglog_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"approx_count_distinct_agg\",\n    requires = [\n        approx_count_distinct_trans,\n        hyperloglog_final,\n        hyperloglog_combine,\n        hyperloglog_serialize,\n        hyperloglog_deserialize\n    ],\n);\n\n#[pg_extern(immutable, parallel_safe)]\npub fn hyperloglog_union<'a>(\n    state: Internal,\n    other: Option<HyperLogLog<'a>>,\n    fc: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    hyperloglog_union_inner(unsafe { state.to_inner() }, other, fc).internal()\n}\npub fn hyperloglog_union_inner(\n    state: Option<Inner<HyperLogLogTrans>>,\n    other: Option<HyperLogLog>,\n    fc: pg_sys::FunctionCallInfo,\n) -> Option<Inner<HyperLogLogTrans>> {\n    unsafe {\n        in_aggregate_context(fc, || {\n            let other = match other {\n                Some(other) => other,\n                None => {\n                    return state;\n                }\n            };\n            let mut state = match state {\n                Some(state) => state,\n                None => {\n                    let state = HyperLogLogTrans {\n                        logger: unflatten_log(other).into_owned(),\n                    };\n                    return Some(state.into());\n                }\n            };\n            let other = unflatten_log(other);\n            if state.logger.buildhasher.type_id != other.buildhasher.type_id {\n                error!(\"mismatched types\")\n            }\n            // TODO error on mismatched collation?\n            state.logger.merge_in(&other);\n            Some(state)\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(hyperloglog Hyperloglog)\\n\\\n    (\\n\\\n        stype = internal,\\n\\\n        sfunc = hyperloglog_union,\\n\\\n        finalfunc = hyperloglog_final,\\n\\\n        combinefunc = hyperloglog_combine,\\n\\\n        serialfunc = hyperloglog_serialize,\\n\\\n        deserialfunc = hyperloglog_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"hll_rollup\",\n    requires = [\n        hyperloglog_union,\n        hyperloglog_final,\n        hyperloglog_combine,\n        hyperloglog_serialize,\n        hyperloglog_deserialize\n    ],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_hyperloglog_count<'a>(\n    sketch: HyperLogLog<'a>,\n    _accessor: AccessorDistinctCount,\n) -> i64 {\n    hyperloglog_count(sketch)\n}\n\n#[pg_extern(name = \"distinct_count\", immutable, parallel_safe)]\npub fn hyperloglog_count<'a>(hyperloglog: HyperLogLog<'a>) -> i64 {\n    // count does not depend on the type parameters\n    let log = match &hyperloglog.log {\n        Storage::Sparse {\n            num_compressed,\n            precision,\n            compressed,\n            ..\n        } => HLL::<HashableDatum, ()>::from_sparse_parts(\n            compressed.slice(),\n            *num_compressed,\n            *precision,\n            (),\n        ),\n        Storage::Dense {\n            precision,\n            registers,\n            ..\n        } => HLL::<HashableDatum, ()>::from_dense_parts(registers.slice(), *precision, ()),\n    };\n    log.immutable_estimate_count() as i64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_hyperloglog_error<'a>(sketch: HyperLogLog<'a>, _accessor: AccessorStderror) -> f64 {\n    hyperloglog_error(sketch)\n}\n\n#[pg_extern(name = \"stderror\", immutable, parallel_safe)]\npub fn hyperloglog_error<'a>(hyperloglog: HyperLogLog<'a>) -> f64 {\n    let precision = match hyperloglog.log {\n        Storage::Sparse { precision, .. } => precision,\n        Storage::Dense { precision, .. } => precision,\n    };\n    hyperloglogplusplus::error_for_precision(precision)\n}\n\nimpl HyperLogLog<'_> {\n    pub fn build_from(\n        size: i32,\n        type_id: Oid,\n        collation: Option<Oid>,\n        data: impl Iterator<Item = pg_sys::Datum>,\n    ) -> HyperLogLog<'static> {\n        unsafe {\n            let b = TryInto::<usize>::try_into(size)\n                .unwrap()\n                .checked_next_power_of_two()\n                .unwrap()\n                .trailing_zeros();\n            let hasher = DatumHashBuilder::from_type_id(type_id, collation);\n            let mut logger: HLL<HashableDatum, DatumHashBuilder> = HLL::new(b as u8, hasher);\n\n            for datum in data {\n                logger.add(&HashableDatum(datum));\n            }\n\n            flatten_log(&mut logger)\n        }\n    }\n}\n\nfn flatten_log(hyperloglog: &mut HLL<HashableDatum, DatumHashBuilder>) -> HyperLogLog<'static> {\n    let (element_type, collation) = {\n        let hasher = &hyperloglog.buildhasher;\n        (ShortTypeId(hasher.type_id), PgCollationId(hasher.collation))\n    };\n\n    // we need to flatten the vector to a single buffer that contains\n    // both the size, the data, and the varlen header\n\n    let flat = match hyperloglog.to_parts() {\n        HyperLogLogStorage::Sparse(sparse) => unsafe {\n            flatten!(HyperLogLog {\n                log: Storage::Sparse {\n                    element_type,\n                    collation,\n                    num_compressed: sparse.num_compressed,\n                    precision: sparse.precision,\n                    compressed_bytes: sparse.compressed.num_bytes() as u32,\n                    compressed: sparse.compressed.bytes().into(),\n                }\n            })\n        },\n        HyperLogLogStorage::Dense(dense) => unsafe {\n            // TODO check that precision and length match?\n            flatten!(HyperLogLog {\n                log: Storage::Dense {\n                    element_type,\n                    collation,\n                    precision: dense.precision,\n                    registers: dense.registers.bytes().into(),\n                }\n            })\n        },\n    };\n    flat\n}\n\nfn unflatten_log(hyperloglog: HyperLogLog) -> HLL<HashableDatum, DatumHashBuilder> {\n    match &hyperloglog.log {\n        Storage::Sparse {\n            num_compressed,\n            precision,\n            compressed,\n            element_type,\n            collation,\n            compressed_bytes: _,\n        } => HLL::<HashableDatum, DatumHashBuilder>::from_sparse_parts(\n            compressed.slice(),\n            *num_compressed,\n            *precision,\n            unsafe { DatumHashBuilder::from_type_id(element_type.0, Some(collation.0)) },\n        ),\n        Storage::Dense {\n            precision,\n            registers,\n            element_type,\n            collation,\n        } => HLL::<HashableDatum, DatumHashBuilder>::from_dense_parts(\n            registers.slice(),\n            *precision,\n            unsafe { DatumHashBuilder::from_type_id(element_type.0, Some(collation.0)) },\n        ),\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n\n    use pgrx_macros::pg_test;\n    use rand::distributions::{Distribution, Uniform};\n\n    #[pg_test]\n    fn test_hll_aggregate() {\n        Spi::connect_mut(|client| {\n            let text = client\n                .update(\n                    \"SELECT \\\n                        hyperloglog(32, v::float)::TEXT \\\n                        FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                log:Dense(\\\n                    element_type:FLOAT8,\\\n                    collation:None,\\\n                    precision:5,\\\n                    registers:[\\\n                        20,64,132,12,81,1,8,64,133,4,64,136,4,82,3,12,17,\\\n                        65,24,32,197,16,32,132,255\\\n                    ]\\\n                )\\\n            )\";\n            assert_eq!(text.unwrap(), expected);\n\n            let (count, arrow_count) = client\n                .update(\n                    \"SELECT \\\n                    distinct_count(\\\n                        hyperloglog(32, v::float)\\\n                    ), \\\n                    hyperloglog(32, v::float) -> distinct_count() \\\n                    FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<i64, i64>()\n                .unwrap();\n            assert_eq!(count, Some(132));\n            assert_eq!(count, arrow_count);\n\n            let count2 = client\n                .update(&format!(\"SELECT distinct_count('{expected}')\"), None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(count2, count);\n        });\n    }\n\n    #[pg_test]\n    // Should have same results as test_hll_distinct_aggregate running with the same number of buckets\n    fn test_approx_count_distinct_aggregate() {\n        Spi::connect_mut(|client| {\n            let text = client\n                .update(\n                    \"SELECT \\\n                        approx_count_distinct(v::float)::TEXT \\\n                        FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                log:Sparse(\\\n                    num_compressed:100,\\\n                    element_type:FLOAT8,\\\n                    collation:None,\\\n                    compressed_bytes:320,\\\n                    precision:15,\\\n                    compressed:[\\\n                    4,61,17,164,87,15,68,239,255,132,121,35,164,5,74,132,160,\\\n                    109,4,177,61,100,68,200,4,144,32,132,118,9,228,190,94,68,\\\n                    120,56,36,121,213,200,97,65,3,200,108,96,2,72,128,10,2,100,\\\n                    182,161,36,218,115,196,202,145,228,189,224,132,21,63,36,\\\n                    88,116,100,162,122,132,139,97,228,245,19,36,242,15,228,115,\\\n                    65,164,114,2,8,224,32,2,72,157,130,2,68,232,93,136,105,1,2,\\\n                    132,16,59,4,34,46,8,244,104,2,226,240,8,82,159,2,200,225,49,\\\n                    2,132,96,9,4,222,195,164,54,22,228,201,59,164,168,27,100,32,\\\n                    58,8,76,32,2,36,56,17,136,18,143,4,132,162,156,196,178,22,\\\n                    132,119,72,228,213,48,4,26,63,68,28,156,36,151,75,36,19,202,\\\n                    164,152,111,164,177,240,98,27,196,254,46,8,138,82,6,164,53,38,\\\n                    36,125,151,8,167,213,3,4,167,248,68,183,61,36,149,32,164,112,\\\n                    121,164,14,139,100,56,166,164,24,48,8,33,90,2,132,115,89,72,\\\n                    100,112,5,196,221,128,228,245,33,4,216,92,8,33,195,6,100,8,54,\\\n                    200,74,2,5,200,101,158,3,228,106,110,72,151,98,2,228,38,26,196,\\\n                    143,15,36,122,57,200,191,43,2,164,225,186,196,219,46,36,26,146,\\\n                    228,129,128,136,6,183,2,4,238,106,200,48,168,2,164,14,13,68,55,\\\n                    196,132,208,90,164,50,130,68,58,137,196,3,88,196,71,31\\\n                    ]\\\n                )\\\n            )\";\n            assert_eq!(text.unwrap(), expected);\n\n            let (count, arrow_count) = client\n                .update(\n                    \"SELECT \\\n                    distinct_count(\\\n                        approx_count_distinct(v::float)\\\n                    ), \\\n                    approx_count_distinct(v::float) -> distinct_count() \\\n                    FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<i64, i64>()\n                .unwrap();\n            assert_eq!(count, Some(100));\n            assert_eq!(count, arrow_count);\n\n            let count2 = client\n                .update(&format!(\"SELECT distinct_count('{expected}')\"), None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(count2, count);\n        });\n    }\n\n    #[pg_test]\n    fn test_hll_byte_io() {\n        unsafe {\n            // Unable to build the hyperloglog through hyperloglog_trans, as that requires a valid fcinfo to determine OIDs.\n\n            let hasher = DatumHashBuilder::from_type_id(\n                pg_sys::TEXTOID,\n                Some(crate::serialization::collations::DEFAULT_COLLATION_OID),\n            );\n            let mut control = HyperLogLogTrans {\n                logger: HLL::new(6, hasher),\n            };\n            control.logger.add(&HashableDatum(\n                rust_str_to_text_p(\"first\").into_datum().unwrap(),\n            ));\n            control.logger.add(&HashableDatum(\n                rust_str_to_text_p(\"second\").into_datum().unwrap(),\n            ));\n            control.logger.add(&HashableDatum(\n                rust_str_to_text_p(\"first\").into_datum().unwrap(),\n            ));\n            control.logger.add(&HashableDatum(\n                rust_str_to_text_p(\"second\").into_datum().unwrap(),\n            ));\n            control.logger.add(&HashableDatum(\n                rust_str_to_text_p(\"third\").into_datum().unwrap(),\n            ));\n\n            let buffer = hyperloglog_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = pgrx::varlena::varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let mut expected = vec![\n                1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 0, 0, 0, 0, 136, 136, 9, 7,\n                8, 74, 76, 47, 200, 231, 53, 25, 3, 0, 0, 0, 0, 0, 0, 0, 6, 9, 0, 0, 0, 1,\n            ];\n            bincode::serialize_into(\n                &mut expected,\n                &PgCollationId(crate::serialization::collations::DEFAULT_COLLATION_OID),\n            )\n            .unwrap();\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let new_state =\n                hyperloglog_deserialize_inner(bytea(pg_sys::Datum::from(expected.as_ptr())));\n\n            control.logger.merge_all(); // Sparse representation buffers always merged on serialization\n            assert!(*new_state == control);\n\n            // Now generate a dense represenataion and validate that\n            for i in 0..500 {\n                control.logger.add(&HashableDatum(\n                    rust_str_to_text_p(&i.to_string()).into_datum().unwrap(),\n                ));\n            }\n\n            let buffer = hyperloglog_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = pgrx::varlena::varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let mut expected = vec![\n                1, 1, 1, 0, 0, 0, 49, 0, 0, 0, 0, 0, 0, 0, 20, 65, 2, 12, 48, 199, 20, 33, 4, 12,\n                49, 67, 16, 81, 66, 32, 145, 131, 24, 49, 4, 20, 33, 5, 8, 81, 66, 12, 81, 4, 8,\n                49, 2, 8, 65, 131, 24, 32, 133, 12, 50, 66, 12, 48, 197, 12, 81, 130, 255, 58, 6,\n                255, 255, 255, 255, 255, 255, 255, 3, 9, 0, 0, 0, 1,\n            ];\n            bincode::serialize_into(\n                &mut expected,\n                &PgCollationId(crate::serialization::collations::DEFAULT_COLLATION_OID),\n            )\n            .unwrap();\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let new_state =\n                hyperloglog_deserialize_inner(bytea(pg_sys::Datum::from(expected.as_ptr())));\n\n            assert!(*new_state == control);\n        }\n    }\n\n    #[pg_test]\n    fn test_hll_aggregate_int() {\n        Spi::connect_mut(|client| {\n            let text = client\n                .update(\n                    \"SELECT hyperloglog(32, v::int)::TEXT\n                    FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                log:Dense(\\\n                    element_type:INT4,\\\n                    collation:None,\\\n                    precision:5,\\\n                    registers:[\\\n                        8,49,0,12,32,129,24,32,195,16,33,2,12,1,68,4,16,\\\n                        196,20,64,133,8,17,67,255\\\n                    ]\\\n                )\\\n            )\";\n            assert_eq!(text.unwrap(), expected);\n\n            let count = client\n                .update(\n                    \"SELECT \\\n                distinct_count(\\\n                    hyperloglog(32, v::int)\\\n                ) FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(count, Some(96));\n\n            let count2 = client\n                .update(&format!(\"SELECT distinct_count('{expected}')\"), None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(count2, count);\n        });\n    }\n\n    #[pg_test]\n    fn test_hll_aggregate_text() {\n        Spi::connect_mut(|client| {\n            use crate::serialization::PgCollationId;\n\n            let text = client\n                .update(\n                    \"SELECT \\\n                        hyperloglog(32, v::text)::TEXT \\\n                    FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let default_collation = ron::to_string(&PgCollationId(\n                crate::serialization::collations::DEFAULT_COLLATION_OID,\n            ))\n            .unwrap();\n            let expected = format!(\n                \"(\\\n                version:1,\\\n                log:Dense(\\\n                    element_type:TEXT,\\\n                    collation:{default_collation},\\\n                    precision:5,\\\n                    registers:[\\\n                        12,33,3,8,33,4,20,50,3,12,32,133,4,32,67,8,48,\\\n                        128,8,33,4,8,32,197,255\\\n                    ]\\\n                )\\\n            )\"\n            );\n            assert_eq!(text.unwrap(), expected);\n\n            let count = client\n                .update(\n                    \"SELECT distinct_count(\\\n                    hyperloglog(32, v::text)\\\n                ) FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(count, Some(111));\n\n            let count2 = client\n                .update(&format!(\"SELECT distinct_count('{expected}')\"), None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(count2, count);\n        });\n    }\n\n    #[pg_test]\n    fn test_hll_union_text() {\n        Spi::connect_mut(|client| {\n            {\n                // self-union should be a nop\n                let expected = client\n                    .update(\n                        \"SELECT \\\n                                hyperloglog(32, v::text)::TEXT \\\n                            FROM generate_series(1, 100) v\",\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<String>()\n                    .unwrap()\n                    .unwrap();\n\n                let text = client\n                    .update(\n                        \"SELECT rollup(logs)::text \\\n                        FROM (\\\n                            (SELECT hyperloglog(32, v::text) logs \\\n                             FROM generate_series(1, 100) v\\\n                            ) UNION ALL \\\n                            (SELECT hyperloglog(32, v::text) \\\n                             FROM generate_series(1, 100) v)\\\n                        ) q\",\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<String>()\n                    .unwrap();\n\n                assert_eq!(text.unwrap(), expected);\n            }\n\n            {\n                // differing unions should be a sum of the distinct counts\n                let query = \"SELECT distinct_count(rollup(logs)) \\\n                    FROM (\\\n                        (SELECT hyperloglog(32, v::text) logs \\\n                         FROM generate_series(1, 100) v) \\\n                        UNION ALL \\\n                        (SELECT hyperloglog(32, v::text) \\\n                         FROM generate_series(50, 150) v)\\\n                    ) q\";\n                let count = client\n                    .update(query, None, &[])\n                    .unwrap()\n                    .first()\n                    .get_one::<i64>()\n                    .unwrap();\n\n                assert_eq!(count, Some(153));\n            }\n        });\n    }\n\n    #[pg_test]\n    fn test_hll_null_input_yields_null_output() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\"SELECT hyperloglog(32, null::int)::TEXT\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(output, None)\n        })\n    }\n\n    #[pg_test(\n        error = \"Invalid value for size 2. Size must be between 16 and 262144, though less than 1024 not recommended\"\n    )]\n    fn test_hll_error_too_small() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\"SELECT hyperloglog(2, 'foo'::text)::TEXT\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(output, None)\n        })\n    }\n\n    #[pg_test]\n    fn test_hll_size_min() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\"SELECT hyperloglog(16, 'foo'::text)::TEXT\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert!(output.is_some())\n        })\n    }\n\n    #[pg_test]\n    fn test_hll_size_max() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\"SELECT hyperloglog(262144, 'foo'::text)::TEXT\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert!(output.is_some())\n        })\n    }\n\n    #[pg_test]\n    fn stderror_arrow_match() {\n        Spi::connect_mut(|client| {\n            let (count, arrow_count) = client\n                .update(\n                    \"SELECT \\\n                    stderror(\\\n                        hyperloglog(32, v::float)\\\n                    ), \\\n                    hyperloglog(32, v::float) -> stderror() \\\n                    FROM generate_series(1, 100) v\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n            assert_eq!(Some(0.18384776310850234), count);\n            assert_eq!(count, arrow_count);\n        });\n    }\n\n    #[pg_test]\n    fn bias_correct_values_accurate() {\n        const NUM_BIAS_TRIALS: usize = 5;\n        const MAX_TRIAL_ERROR: f64 = 0.05;\n        Spi::connect_mut(|client| {\n            // This should match THRESHOLD_DATA_VEC from b=12-18\n            let thresholds = [3100, 6500, 11500, 20000, 50000, 120000, 350000];\n            let rand_precision: Uniform<usize> = Uniform::new_inclusive(12, 18);\n            let mut rng = rand::thread_rng();\n            for _ in 0..NUM_BIAS_TRIALS {\n                let precision = rand_precision.sample(&mut rng);\n                let rand_cardinality: Uniform<usize> =\n                    Uniform::new_inclusive(thresholds[precision - 12], 5 * (1 << precision));\n                let cardinality = rand_cardinality.sample(&mut rng);\n                let query = format!(\n                    \"SELECT hyperloglog({}, v) -> distinct_count() FROM generate_series(1, {}) v\",\n                    1 << precision,\n                    cardinality\n                );\n\n                let estimate = client\n                    .update(&query, None, &[])\n                    .unwrap()\n                    .first()\n                    .get_one::<i64>()\n                    .unwrap()\n                    .unwrap();\n\n                let error = (estimate as f64 / cardinality as f64).abs() - 1.;\n                assert!(error < MAX_TRIAL_ERROR, \"hyperloglog with {} buckets on cardinality {} gave a result of {}.  Resulting error {} exceeds max allowed ({})\", 2^precision, cardinality, estimate, error, MAX_TRIAL_ERROR);\n            }\n        });\n    }\n\n    #[pg_test(\n        error = \"Invalid value for size 262145. Size must be between 16 and 262144, though less than 1024 not recommended\"\n    )]\n    fn test_hll_error_too_large() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\"SELECT hyperloglog(262145, 'foo'::text)::TEXT\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(output, None)\n        })\n    }\n\n    #[pg_test]\n    fn test_hll_null_rollup() {\n        Spi::connect_mut(|client| {\n            let output1 = client\n                .update(\n                    \"SELECT distinct_count(rollup(logs))\n                FROM (\n                    (SELECT hyperloglog(16, v::text) logs FROM generate_series(1, 5) v)\n                    UNION ALL\n                    (SELECT hyperloglog(16, v::text) FROM generate_series(6, 10) v WHERE v <=5)\n                ) hll;\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n\n            let output2 = client\n                .update(\n                    \"SELECT distinct_count(rollup(logs))\n                FROM (\n                    (SELECT hyperloglog(16, v::text) logs FROM generate_series(1, 5) v)\n                ) hll;\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n\n            assert_eq!(output1, output2);\n        })\n    }\n\n    //TODO test continuous aggregates\n}\n"
  },
  {
    "path": "extension/src/lib.rs",
    "content": "// so we can support upgrading pgrx\n#![allow(unexpected_cfgs)]\n// so we can allow very new Clippy lints\n#![allow(unknown_lints)]\n// flat_serialize! alignment checks hit this for any single byte field (of which all pg_types! have two by default)\n#![allow(clippy::modulo_one)]\n// some disagreement between clippy and the rust compiler about when lifetime are and are not needed\n#![allow(clippy::extra_unused_lifetimes)]\n// every function calling in_aggregate_context should be unsafe\n#![allow(clippy::not_unsafe_ptr_arg_deref)]\n// since 0.5 pgrx requires non-elided lifetimes on extern functions: https://github.com/tcdi/pgrx/issues/721\n#![allow(clippy::needless_lifetimes)]\n// triggered by pg_extern macros\n#![allow(clippy::useless_conversion)]\n// caused by pgrx\n#![allow(clippy::unnecessary_lazy_evaluations)]\n// clippy triggers an internal complier error checking this\n#![allow(clippy::unnecessary_literal_unwrap)]\n\npub mod accessors;\npub mod asap;\npub mod candlestick;\npub mod counter_agg;\npub mod countminsketch;\npub mod frequency;\npub mod gauge_agg;\npub mod heartbeat_agg;\npub mod hyperloglog;\npub mod lttb;\npub mod nmost;\npub mod range;\npub mod saturation;\npub(crate) mod serialization;\npub mod state_aggregate;\npub mod stats_agg;\npub mod tdigest;\npub mod time_vector;\npub mod time_weighted_average;\npub mod uddsketch;\npub mod utilities;\n\nmod aggregate_utils;\nmod datum_utils;\nmod duration;\nmod palloc;\nmod pg_any_element;\nmod raw;\nmod stabilization_info;\nmod stabilization_tests;\n\n#[macro_use]\nmod type_builder;\n\n#[cfg(any(test, feature = \"pg_test\"))]\nmod aggregate_builder_tests;\n\nuse pgrx::*;\n\npgrx::pg_module_magic!();\n\n#[pg_guard]\npub extern \"C-unwind\" fn _PG_init() {\n    // Nothing to do here\n}\n\nextension_sql!(\n    r#\"GRANT USAGE ON SCHEMA toolkit_experimental TO PUBLIC;\"#,\n    name = \"final_grant\",\n    finalize,\n);\n\n#[cfg(test)]\npub mod pg_test {\n    pub fn setup(_options: Vec<&str>) {\n        // perform one-off initialization when the pg_test framework starts\n    }\n\n    pub fn postgresql_conf_options() -> Vec<&'static str> {\n        // return any postgresql.conf settings that are required for your tests\n        vec![]\n    }\n}\n"
  },
  {
    "path": "extension/src/lttb.rs",
    "content": "use pgrx::*;\nuse std::borrow::Cow;\n\nuse crate::{\n    aggregate_utils::in_aggregate_context,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    time_vector,\n};\n\nuse tspoint::TSPoint;\n\nuse crate::time_vector::{Timevector_TSTZ_F64, Timevector_TSTZ_F64Data};\n\npub struct LttbTrans {\n    series: Vec<TSPoint>,\n    resolution: usize,\n    gap_interval: i64,\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn lttb_trans(\n    state: Internal,\n    time: crate::raw::TimestampTz,\n    val: Option<f64>,\n    resolution: i32,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    lttb_trans_inner(unsafe { state.to_inner() }, time, val, resolution, fcinfo).internal()\n}\npub fn lttb_trans_inner(\n    state: Option<Inner<LttbTrans>>,\n    time: crate::raw::TimestampTz,\n    val: Option<f64>,\n    resolution: i32,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<LttbTrans>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let val = match val {\n                None => return state,\n                Some(val) => val,\n            };\n            let mut state = match state {\n                Some(state) => state,\n                None => {\n                    if resolution <= 2 {\n                        error!(\"resolution must be greater than 2\")\n                    }\n                    LttbTrans {\n                        series: vec![],\n                        resolution: resolution as usize,\n                        gap_interval: 0,\n                    }\n                    .into()\n                }\n            };\n\n            state.series.push(TSPoint {\n                ts: time.into(),\n                val,\n            });\n            Some(state)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn lttb_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    lttb_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\npub fn lttb_final_inner(\n    state: Option<Inner<LttbTrans>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state,\n            };\n            state.series.sort_by_key(|point| point.ts);\n            let downsampled = lttb(&state.series[..], state.resolution);\n            flatten!(Timevector_TSTZ_F64 {\n                num_points: downsampled.len() as u32,\n                flags: time_vector::FLAG_IS_SORTED,\n                internal_padding: [0; 3],\n                points: (&*downsampled).into(),\n                null_val: std::vec::from_elem(0_u8, downsampled.len().div_ceil(8)).into()\n            })\n            .into()\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn gp_lttb_trans(\n    state: Internal,\n    time: crate::raw::TimestampTz,\n    val: Option<f64>,\n    gap: crate::raw::Interval,\n    resolution: i32,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let state = unsafe { state.to_inner() };\n    let needs_interval = state.is_none();\n\n    // Don't love this code, but need to compute gap_val if needed before time is moved\n    let gap_val = if needs_interval {\n        crate::datum_utils::interval_to_ms(&time, &gap)\n    } else {\n        0\n    };\n\n    let mut trans = lttb_trans_inner(state, time, val, resolution, fcinfo);\n    if needs_interval {\n        #[allow(clippy::manual_inspect)] // need to mutate s\n        trans.as_mut().map(|s| {\n            s.gap_interval = gap_val;\n            s\n        });\n    }\n    trans.internal()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn gp_lttb_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    gap_preserving_lttb_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\npub fn gap_preserving_lttb_final_inner(\n    state: Option<Inner<LttbTrans>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state,\n            };\n            state.series.sort_by_key(|point| point.ts);\n\n            let count = state.series.len();\n            let max_gap = if state.gap_interval > 0 {\n                state.gap_interval\n            } else {\n                let range = state.series[count - 1].ts - state.series[0].ts;\n                range / state.resolution as i64\n            };\n\n            // Tracking endpoints remaining will keep us from assigning too many points\n            // to early LTTB computations when there are lots of gaps later in the timeseries\n            let mut endpoints_remaining = 2;\n            let mut start = 0;\n            for i in 0..count - 1 {\n                if state.series[i + 1].ts - state.series[i].ts > max_gap {\n                    if i == start {\n                        endpoints_remaining += 1;\n                    } else {\n                        endpoints_remaining += 2;\n                    }\n                    start = i + 1;\n                }\n            }\n\n            let mut points_remaining = state.resolution as i64;\n            let mut downsampled = vec![];\n            start = 0;\n\n            for i in 0..count - 1 {\n                if state.series[i + 1].ts - state.series[i].ts > max_gap {\n                    if i == start {\n                        // 1 len subarray\n                        downsampled.push(state.series[i]);\n                        start = i + 1;\n                        points_remaining -= 1;\n                        endpoints_remaining -= 1;\n                    } else {\n                        let sgmt_pct_of_remaining_pts =\n                            (i - start - 1) as f64 / (count - start - endpoints_remaining) as f64;\n                        let pts_for_sgmt = std::cmp::max(\n                            ((points_remaining - endpoints_remaining as i64) as f64\n                                * sgmt_pct_of_remaining_pts) as usize,\n                            0,\n                        ) + 2;\n                        downsampled\n                            .append(&mut lttb(&state.series[start..=i], pts_for_sgmt).into_owned());\n                        start = i + 1;\n                        points_remaining -= pts_for_sgmt as i64;\n                        endpoints_remaining -= 2;\n                    }\n                }\n            }\n            // remainder\n            if start == count - 1 {\n                downsampled.push(state.series[count - 1]);\n            } else {\n                downsampled.append(\n                    &mut lttb(\n                        &state.series[start..count],\n                        std::cmp::max(points_remaining, 2) as usize,\n                    )\n                    .into_owned(),\n                );\n            }\n\n            flatten!(Timevector_TSTZ_F64 {\n                num_points: downsampled.len() as u32,\n                flags: time_vector::FLAG_IS_SORTED,\n                internal_padding: [0; 3],\n                null_val: std::vec::from_elem(0_u8, downsampled.len().div_ceil(8)).into(),\n                points: downsampled.into(),\n            })\n            .into()\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\nCREATE AGGREGATE lttb(ts TIMESTAMPTZ, value DOUBLE PRECISION, resolution integer) (\\n\\\n    sfunc = lttb_trans,\\n\\\n    stype = internal,\\n\\\n    finalfunc = lttb_final\\n\\\n);\\n\\\n\",\n    name = \"lttb_agg\",\n    requires = [lttb_trans, lttb_final],\n);\n\nextension_sql!(\"\\n\\\nCREATE AGGREGATE toolkit_experimental.gp_lttb(ts TIMESTAMPTZ, value DOUBLE PRECISION, resolution integer) (\\n\\\n    sfunc = lttb_trans,\\n\\\n    stype = internal,\\n\\\n    finalfunc = toolkit_experimental.gp_lttb_final\\n\\\n);\\n\\\n\",\nname = \"gp_lttb_agg\",\nrequires = [lttb_trans, gp_lttb_final],\n);\n\nextension_sql!(\"\\n\\\nCREATE AGGREGATE toolkit_experimental.gp_lttb(ts TIMESTAMPTZ, value DOUBLE PRECISION, gapsize INTERVAL, resolution integer) (\\n\\\n    sfunc = toolkit_experimental.gp_lttb_trans,\\n\\\n    stype = internal,\\n\\\n    finalfunc = toolkit_experimental.gp_lttb_final\\n\\\n);\\n\\\n\",\nname = \"gp_lttb_agg_with_size\",\nrequires = [gp_lttb_trans, gp_lttb_final],\n);\n\n// based on https://github.com/jeromefroe/lttb-rs version 0.2.0\npub fn lttb(data: &[TSPoint], threshold: usize) -> Cow<'_, [TSPoint]> {\n    if threshold >= data.len() || threshold == 0 {\n        // Nothing to do.\n        return Cow::Borrowed(data);\n    }\n\n    let mut sampled = Vec::with_capacity(threshold);\n\n    // Bucket size. Leave room for start and end data points.\n    let every = ((data.len() - 2) as f64) / ((threshold - 2) as f64);\n\n    // Initially a is the first point in the triangle.\n    let mut a = 0;\n\n    // Always add the first point.\n    sampled.push(data[a]);\n\n    for i in 0..threshold - 2 {\n        // Calculate point average for next bucket (containing c).\n        let mut avg_x = 0i64;\n        let mut avg_y = 0f64;\n\n        let avg_range_start = (((i + 1) as f64) * every) as usize + 1;\n\n        let mut end = (((i + 2) as f64) * every) as usize + 1;\n        if end >= data.len() {\n            end = data.len();\n        }\n        let avg_range_end = end;\n\n        let avg_range_length = (avg_range_end - avg_range_start) as f64;\n\n        for i in 0..(avg_range_end - avg_range_start) {\n            let idx = avg_range_start + i;\n            avg_x += data[idx].ts;\n            avg_y += data[idx].val;\n        }\n        avg_x /= avg_range_length as i64;\n        avg_y /= avg_range_length;\n\n        // Get the range for this bucket.\n        let range_offs = ((i as f64) * every) as usize + 1;\n        let range_to = (((i + 1) as f64) * every) as usize + 1;\n\n        // Point a.\n        let point_a_x = data[a].ts;\n        let point_a_y = data[a].val;\n\n        let mut max_area = -1f64;\n        let mut next_a = range_offs;\n        for i in 0..(range_to - range_offs) {\n            let idx = range_offs + i;\n\n            // Calculate triangle area over three buckets.\n            let area = ((point_a_x - avg_x) as f64 * (data[idx].val - point_a_y)\n                - (point_a_x - data[idx].ts) as f64 * (avg_y - point_a_y))\n                .abs()\n                * 0.5;\n            if area > max_area {\n                max_area = area;\n                next_a = idx; // Next a is this b.\n            }\n        }\n\n        sampled.push(data[next_a]); // Pick this point from the bucket.\n        a = next_a; // This a is the next a (chosen b).\n    }\n\n    // Always add the last point.\n    sampled.push(data[data.len() - 1]);\n\n    Cow::Owned(sampled)\n}\n\n#[pg_extern(name = \"lttb\", immutable, parallel_safe)]\npub fn lttb_on_timevector(\n    series: Timevector_TSTZ_F64<'static>,\n    threshold: i32,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    lttb_ts(series, threshold as usize).into()\n}\n\n// based on https://github.com/jeromefroe/lttb-rs version 0.2.0\npub fn lttb_ts(data: Timevector_TSTZ_F64, threshold: usize) -> Timevector_TSTZ_F64 {\n    if !data.is_sorted() {\n        panic!(\"lttb requires sorted timevector\");\n    }\n\n    if threshold >= data.num_points() || threshold == 0 {\n        // Nothing to do.\n        return data.in_current_context(); // can we avoid this copy???\n    }\n\n    let mut sampled = Vec::with_capacity(threshold);\n\n    // Bucket size. Leave room for start and end data points.\n    let every = ((data.num_points() - 2) as f64) / ((threshold - 2) as f64);\n\n    // Initially a is the first point in the triangle.\n    let mut a = 0;\n\n    // Always add the first point.\n    sampled.push(data.get(a).unwrap());\n\n    for i in 0..threshold - 2 {\n        // Calculate point average for next bucket (containing c).\n        let mut avg_x = 0i64;\n        let mut avg_y = 0f64;\n\n        let avg_range_start = (((i + 1) as f64) * every) as usize + 1;\n\n        let mut end = (((i + 2) as f64) * every) as usize + 1;\n        if end >= data.num_points() {\n            end = data.num_points();\n        }\n        let avg_range_end = end;\n\n        let avg_range_length = (avg_range_end - avg_range_start) as f64;\n\n        for i in 0..(avg_range_end - avg_range_start) {\n            let idx = avg_range_start + i;\n            let point = data.get(idx).unwrap();\n            avg_x += point.ts;\n            avg_y += point.val;\n        }\n        avg_x /= avg_range_length as i64;\n        avg_y /= avg_range_length;\n\n        // Get the range for this bucket.\n        let range_offs = ((i as f64) * every) as usize + 1;\n        let range_to = (((i + 1) as f64) * every) as usize + 1;\n\n        // Point a.\n        let point_a_x = data.get(a).unwrap().ts;\n        let point_a_y = data.get(a).unwrap().val;\n\n        let mut max_area = -1f64;\n        let mut next_a = range_offs;\n        for i in 0..(range_to - range_offs) {\n            let idx = range_offs + i;\n\n            // Calculate triangle area over three buckets.\n            let area = ((point_a_x - avg_x) as f64 * (data.get(idx).unwrap().val - point_a_y)\n                - (point_a_x - data.get(idx).unwrap().ts) as f64 * (avg_y - point_a_y))\n                .abs()\n                * 0.5;\n            if area > max_area {\n                max_area = area;\n                next_a = idx; // Next a is this b.\n            }\n        }\n\n        sampled.push(data.get(next_a).unwrap()); // Pick this point from the bucket.\n        a = next_a; // This a is the next a (chosen b).\n    }\n\n    // Always add the last point.\n    sampled.push(data.get(data.num_points() - 1).unwrap());\n\n    let nulls_len = sampled.len().div_ceil(8);\n\n    crate::build! {\n        Timevector_TSTZ_F64 {\n            num_points: sampled.len() as _,\n            flags: time_vector::FLAG_IS_SORTED,\n            internal_padding: [0; 3],\n            points: sampled.into(),\n            null_val: std::vec::from_elem(0_u8, nulls_len).into(),\n        }\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_lttb_equivalence() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE test(time TIMESTAMPTZ, value DOUBLE PRECISION);\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\n                \"INSERT INTO test\n                SELECT time, value\n                FROM toolkit_experimental.generate_periodic_normal_series('2020-01-01 UTC'::timestamptz, NULL);\", None, &[]).unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE results1(time TIMESTAMPTZ, value DOUBLE PRECISION);\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO results1\n                SELECT time, value\n                FROM unnest(\n                    (SELECT lttb(time, value, 100) FROM test)\n                );\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE results2(time TIMESTAMPTZ, value DOUBLE PRECISION);\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO results2\n                SELECT time, value\n                FROM unnest(\n                    (SELECT lttb(\n                        (SELECT timevector(time, value) FROM test), 100)\n                    )\n                );\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let delta = client\n                .update(\"SELECT count(*)  FROM results1 r1 FULL OUTER JOIN results2 r2 ON r1 = r2 WHERE r1 IS NULL OR r2 IS NULL;\" , None, &[])\n                .unwrap().first()\n                .get_one::<i64>().unwrap();\n            assert_eq!(delta.unwrap(), 0);\n        })\n    }\n\n    #[pg_test]\n    fn test_lttb_result() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            let mut result = client\n                .update(\n                    r#\"SELECT unnest(lttb(ts, val, 5))::TEXT\n                FROM (VALUES\n                    ('2020-1-1'::timestamptz, 10),\n                    ('2020-1-2'::timestamptz, 21),\n                    ('2020-1-3'::timestamptz, 19),\n                    ('2020-1-4'::timestamptz, 32),\n                    ('2020-1-5'::timestamptz, 12),\n                    ('2020-1-6'::timestamptz, 14),\n                    ('2020-1-7'::timestamptz, 18),\n                    ('2020-1-8'::timestamptz, 29),\n                    ('2020-1-9'::timestamptz, 23),\n                    ('2020-1-10'::timestamptz, 27),\n                    ('2020-1-11'::timestamptz, 14)\n                ) AS v(ts, val)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",10)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",32)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",12)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-08 00:00:00+00\\\",29)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-11 00:00:00+00\\\",14)\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    fn test_gp_lttb() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            let mut result = client\n                .update(\n                    r#\"SELECT unnest(toolkit_experimental.gp_lttb(ts, val, 7))::TEXT\n                FROM (VALUES\n                    ('2020-1-1'::timestamptz, 10),\n                    ('2020-1-2'::timestamptz, 21),\n                    ('2020-1-3'::timestamptz, 19),\n                    ('2020-1-4'::timestamptz, 32),\n                    ('2020-1-5'::timestamptz, 12),\n                    ('2020-2-6'::timestamptz, 14),\n                    ('2020-3-7'::timestamptz, 18),\n                    ('2020-3-8'::timestamptz, 29),\n                    ('2020-3-9'::timestamptz, 23),\n                    ('2020-3-10'::timestamptz, 27),\n                    ('2020-3-11'::timestamptz, 14)\n                ) AS v(ts, val)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",10)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",32)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",12)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-02-06 00:00:00+00\\\",14)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-07 00:00:00+00\\\",18)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-08 00:00:00+00\\\",29)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-11 00:00:00+00\\\",14)\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    fn test_gp_lttb_with_gap() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            let mut result = client\n                .update(\n                    r#\"SELECT unnest(toolkit_experimental.gp_lttb(ts, val, '36hr', 5))::TEXT\n                FROM (VALUES\n                    ('2020-1-1'::timestamptz, 10),\n                    ('2020-1-2'::timestamptz, 21),\n                    ('2020-1-4'::timestamptz, 32),\n                    ('2020-1-5'::timestamptz, 12),\n                    ('2020-2-6'::timestamptz, 14),\n                    ('2020-3-7'::timestamptz, 18),\n                    ('2020-3-8'::timestamptz, 29),\n                    ('2020-3-10'::timestamptz, 27),\n                    ('2020-3-11'::timestamptz, 14)\n                ) AS v(ts, val)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // This should include everything, despite target resolution of 5\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",10)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-02 00:00:00+00\\\",21)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",32)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",12)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-02-06 00:00:00+00\\\",14)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-07 00:00:00+00\\\",18)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-08 00:00:00+00\\\",29)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-10 00:00:00+00\\\",27)\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-03-11 00:00:00+00\\\",14)\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/max_by_float.rs",
    "content": "use pgrx::{iter::TableIterator, *};\n\nuse crate::nmost::max_float::*;\nuse crate::nmost::*;\n\nuse crate::{\n    build, flatten,\n    palloc::{Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\nuse ordered_float::NotNan;\nuse std::cmp::Reverse;\n\ntype MaxByFloatTransType = NMostByTransState<Reverse<NotNan<f64>>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MaxByFloats<'input> {\n        values: MaxFloatsData<'input>,  // Nesting pg_types adds 8 bytes of header\n        data: DatumStore<'input>,\n    }\n}\nron_inout_funcs!(MaxByFloats<'input>);\n\nimpl<'input> From<MaxByFloatTransType> for MaxByFloats<'input> {\n    fn from(item: MaxByFloatTransType) -> Self {\n        let (capacity, val_ary, data) = item.into_sorted_parts();\n        unsafe {\n            flatten!(MaxByFloats {\n                values: build!(MaxFloats {\n                    capacity: capacity as u32,\n                    elements: val_ary.len() as u32,\n                    values: val_ary\n                        .into_iter()\n                        .map(|x| f64::from(x.0))\n                        .collect::<Vec<f64>>()\n                        .into()\n                })\n                .0,\n                data,\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_float_trans(\n    state: Internal,\n    value: f64,\n    data: AnyElement,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_trans_function(\n        unsafe { state.to_inner::<MaxByFloatTransType>() },\n        Reverse(NotNan::new(value).unwrap()),\n        data,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_float_rollup_trans(\n    state: Internal,\n    value: MaxByFloats<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<Reverse<NotNan<f64>>> = value\n        .values\n        .values\n        .clone()\n        .into_iter()\n        .map(|x| Reverse(NotNan::new(x).unwrap()))\n        .collect();\n    nmost_by_rollup_trans_function(\n        unsafe { state.to_inner::<MaxByFloatTransType>() },\n        &values,\n        &value.data,\n        value.values.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_float_final(state: Internal) -> MaxByFloats<'static> {\n    unsafe { state.to_inner::<MaxByFloatTransType>().unwrap().clone() }.into()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn max_n_by_float_to_values(\n    agg: MaxByFloats<'static>,\n    _dummy: Option<AnyElement>,\n) -> TableIterator<'static, (name!(value, f64), name!(data, AnyElement))> {\n    TableIterator::new(\n        agg.values\n            .values\n            .clone()\n            .into_iter()\n            .zip(agg.data.clone().into_anyelement_iter()),\n    )\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE max_n_by(\\n\\\n        value double precision, data AnyElement, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = max_n_by_float_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = max_n_by_float_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_by_float\",\n    requires = [max_n_by_float_trans, max_n_by_float_final],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        MaxByFloats\\n\\\n    ) (\\n\\\n        sfunc = max_n_by_float_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = max_n_by_float_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_by_float_rollup\",\n    requires = [max_n_by_float_rollup_trans, min_n_by_float_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn max_by_float_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val DOUBLE PRECISION, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}.0/128, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(max_n_by(val, data, 3), NULL::data)::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.7734375,\\\"(0.7734375,3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.765625,\\\"(0.765625,2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.7578125,\\\"(0.7578125,1)\\\")\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let mut result =\n                client.update(\n                    \"WITH aggs as (SELECT category, max_n_by(val, data, 5) as agg from data GROUP BY category)\n                        SELECT into_values(rollup(agg), NULL::data)::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.7734375,\\\"(0.7734375,3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.765625,\\\"(0.765625,2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.7578125,\\\"(0.7578125,1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.75,\\\"(0.75,0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.7421875,\\\"(0.7421875,3)\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/max_by_int.rs",
    "content": "use pgrx::{iter::TableIterator, *};\n\nuse crate::nmost::max_int::*;\nuse crate::nmost::*;\n\nuse crate::{\n    build, flatten,\n    palloc::{Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\nuse std::cmp::Reverse;\n\ntype MaxByIntTransType = NMostByTransState<Reverse<i64>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MaxByInts<'input> {\n        values: MaxIntsData<'input>,  // Nesting pg_types adds 8 bytes of header\n        data: DatumStore<'input>,\n    }\n}\nron_inout_funcs!(MaxByInts<'input>);\n\nimpl<'input> From<MaxByIntTransType> for MaxByInts<'input> {\n    fn from(item: MaxByIntTransType) -> Self {\n        let (capacity, val_ary, data) = item.into_sorted_parts();\n        unsafe {\n            flatten!(MaxByInts {\n                values: build!(MaxInts {\n                    capacity: capacity as u32,\n                    elements: val_ary.len() as u32,\n                    values: val_ary\n                        .into_iter()\n                        .map(|x| x.0)\n                        .collect::<Vec<i64>>()\n                        .into()\n                })\n                .0,\n                data,\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_int_trans(\n    state: Internal,\n    value: i64,\n    data: AnyElement,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_trans_function(\n        unsafe { state.to_inner::<MaxByIntTransType>() },\n        Reverse(value),\n        data,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_int_rollup_trans(\n    state: Internal,\n    value: MaxByInts<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<Reverse<i64>> = value\n        .values\n        .values\n        .clone()\n        .into_iter()\n        .map(Reverse)\n        .collect();\n    nmost_by_rollup_trans_function(\n        unsafe { state.to_inner::<MaxByIntTransType>() },\n        &values,\n        &value.data,\n        value.values.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_int_final(state: Internal) -> MaxByInts<'static> {\n    unsafe { state.to_inner::<MaxByIntTransType>().unwrap().clone() }.into()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn max_n_by_int_to_values(\n    agg: MaxByInts<'static>,\n    _dummy: Option<AnyElement>,\n) -> TableIterator<'static, (name!(value, i64), name!(data, AnyElement))> {\n    TableIterator::new(\n        agg.values\n            .values\n            .clone()\n            .into_iter()\n            .zip(agg.data.clone().into_anyelement_iter()),\n    )\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE max_n_by(\\n\\\n        value bigint, data AnyElement, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = max_n_by_int_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = max_n_by_int_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_by_int\",\n    requires = [max_n_by_int_trans, max_n_by_int_final],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        MaxByInts\\n\\\n    ) (\\n\\\n        sfunc = max_n_by_int_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = max_n_by_int_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_by_int_rollup\",\n    requires = [max_n_by_int_rollup_trans, min_n_by_int_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn max_by_int_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\"CREATE TABLE data(val INT8, category INT)\", None, &[])\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(max_n_by(val, data, 3), NULL::data)::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(99,\\\"(99,3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(98,\\\"(98,2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(97,\\\"(97,1)\\\")\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let mut result =\n                client.update(\n                    \"WITH aggs as (SELECT category, max_n_by(val, data, 5) as agg from data GROUP BY category)\n                        SELECT into_values(rollup(agg), NULL::data)::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(99,\\\"(99,3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(98,\\\"(98,2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(97,\\\"(97,1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(96,\\\"(96,0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(95,\\\"(95,3)\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/max_by_time.rs",
    "content": "use pgrx::{iter::TableIterator, *};\n\nuse crate::nmost::max_time::*;\nuse crate::nmost::*;\n\nuse crate::{\n    build, flatten,\n    palloc::{Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\nuse std::cmp::Reverse;\n\ntype MaxByTimeTransType = NMostByTransState<Reverse<pg_sys::TimestampTz>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MaxByTimes<'input> {\n        values: MaxTimesData<'input>,  // Nesting pg_types adds 8 bytes of header\n        data: DatumStore<'input>,\n    }\n}\nron_inout_funcs!(MaxByTimes<'input>);\n\nimpl<'input> From<MaxByTimeTransType> for MaxByTimes<'input> {\n    fn from(item: MaxByTimeTransType) -> Self {\n        let (capacity, val_ary, data) = item.into_sorted_parts();\n        unsafe {\n            flatten!(MaxByTimes {\n                values: build!(MaxTimes {\n                    capacity: capacity as u32,\n                    elements: val_ary.len() as u32,\n                    values: val_ary\n                        .into_iter()\n                        .map(|x| x.0)\n                        .collect::<Vec<pg_sys::TimestampTz>>()\n                        .into()\n                })\n                .0,\n                data,\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_time_trans(\n    state: Internal,\n    value: crate::raw::TimestampTz,\n    data: AnyElement,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_trans_function(\n        unsafe { state.to_inner::<MaxByTimeTransType>() },\n        Reverse(value.into()),\n        data,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_time_rollup_trans(\n    state: Internal,\n    value: MaxByTimes<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<Reverse<pg_sys::TimestampTz>> = value\n        .values\n        .values\n        .clone()\n        .into_iter()\n        .map(Reverse)\n        .collect();\n    nmost_by_rollup_trans_function(\n        unsafe { state.to_inner::<MaxByTimeTransType>() },\n        &values,\n        &value.data,\n        value.values.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_by_time_final(state: Internal) -> MaxByTimes<'static> {\n    unsafe { state.to_inner::<MaxByTimeTransType>().unwrap().clone() }.into()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn max_n_by_time_to_values(\n    agg: MaxByTimes<'static>,\n    _dummy: Option<AnyElement>,\n) -> TableIterator<\n    'static,\n    (\n        name!(value, crate::raw::TimestampTz),\n        name!(data, AnyElement),\n    ),\n> {\n    TableIterator::new(\n        agg.values\n            .values\n            .clone()\n            .into_iter()\n            .map(crate::raw::TimestampTz::from)\n            .zip(agg.data.clone().into_anyelement_iter()),\n    )\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE max_n_by(\\n\\\n        value timestamptz, data AnyElement, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = max_n_by_time_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = max_n_by_time_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_by_time\",\n    requires = [max_n_by_time_trans, max_n_by_time_final],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        MaxByTimes\\n\\\n    ) (\\n\\\n        sfunc = max_n_by_time_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = max_n_by_time_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_by_time_rollup\",\n    requires = [max_n_by_time_rollup_trans, min_n_by_time_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn max_by_time_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val TIMESTAMPTZ, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client.update(\n                    &format!(\"INSERT INTO data VALUES ('2020-1-1 UTC'::timestamptz + {} * '1d'::interval, {})\", i, i % 4),\n                    None,\n                    &[]\n                ).unwrap();\n            }\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(max_n_by(val, data, 3), NULL::data)::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-09 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-09 00:00:00+00\\\"\\\",3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-08 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-08 00:00:00+00\\\"\\\",2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-07 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-07 00:00:00+00\\\"\\\",1)\\\")\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let mut result =\n                client.update(\n                    \"WITH aggs as (SELECT category, max_n_by(val, data, 5) as agg from data GROUP BY category)\n                        SELECT into_values(rollup(agg), NULL::data)::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-09 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-09 00:00:00+00\\\"\\\",3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-08 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-08 00:00:00+00\\\"\\\",2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-07 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-07 00:00:00+00\\\"\\\",1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-06 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-06 00:00:00+00\\\"\\\",0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-04-05 00:00:00+00\\\",\\\"(\\\"\\\"2020-04-05 00:00:00+00\\\"\\\",3)\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/max_float.rs",
    "content": "use pgrx::{iter::SetOfIterator, *};\n\nuse crate::nmost::*;\n\nuse crate::{\n    accessors::{AccessorIntoArray, AccessorIntoValues},\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\nuse ordered_float::NotNan;\nuse std::cmp::Reverse;\n\ntype MaxFloatTransType = NMostTransState<Reverse<NotNan<f64>>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MaxFloats <'input> {\n        capacity : u32,\n        elements : u32,\n        values : [f64; self.elements],\n    }\n}\nron_inout_funcs!(MaxFloats<'input>);\n\nimpl<'input> From<&mut MaxFloatTransType> for MaxFloats<'input> {\n    fn from(item: &mut MaxFloatTransType) -> Self {\n        let heap = std::mem::take(&mut item.heap);\n        unsafe {\n            flatten!(MaxFloats {\n                capacity: item.capacity as u32,\n                elements: heap.len() as u32,\n                values: heap\n                    .into_sorted_vec()\n                    .into_iter()\n                    .map(|x| f64::from(x.0))\n                    .collect::<Vec<f64>>()\n                    .into()\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_float_trans(\n    state: Internal,\n    value: f64,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_function(\n        unsafe { state.to_inner::<MaxFloatTransType>() },\n        Reverse(NotNan::new(value).unwrap()),\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_float_rollup_trans(\n    state: Internal,\n    value: MaxFloats<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<Reverse<NotNan<f64>>> = value\n        .values\n        .clone()\n        .into_iter()\n        .map(|x| Reverse(NotNan::new(x).unwrap()))\n        .collect();\n    nmost_rollup_trans_function(\n        unsafe { state.to_inner::<MaxFloatTransType>() },\n        &values,\n        value.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_float_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_combine(\n        unsafe { state1.to_inner::<MaxFloatTransType>() },\n        unsafe { state2.to_inner::<MaxFloatTransType>() },\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_float_serialize(state: Internal) -> bytea {\n    let state: Inner<MaxFloatTransType> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_float_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: MaxFloatTransType = crate::do_deserialize!(bytes, MaxFloatTransType);\n    Internal::new(i).into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_float_final(state: Internal) -> MaxFloats<'static> {\n    unsafe { &mut *state.to_inner::<MaxFloatTransType>().unwrap() }.into()\n}\n\n#[pg_extern(name = \"into_array\", immutable, parallel_safe)]\npub fn max_n_float_to_array(agg: MaxFloats<'static>) -> Vec<f64> {\n    agg.values.clone().into_vec()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn max_n_float_to_values(agg: MaxFloats<'static>) -> SetOfIterator<'static, f64> {\n    SetOfIterator::new(agg.values.clone().into_iter())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_float_into_values(\n    agg: MaxFloats<'static>,\n    _accessor: AccessorIntoValues,\n) -> SetOfIterator<'static, f64> {\n    max_n_float_to_values(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_float_into_array(\n    agg: MaxFloats<'static>,\n    _accessor: AccessorIntoArray,\n) -> Vec<f64> {\n    max_n_float_to_array(agg)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE max_n(\\n\\\n        value double precision, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = max_n_float_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = max_n_float_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = max_n_float_serialize,\\n\\\n        deserialfunc = max_n_float_deserialize,\\n\\\n        finalfunc = max_n_float_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_float\",\n    requires = [\n        max_n_float_trans,\n        max_n_float_final,\n        max_n_float_combine,\n        max_n_float_serialize,\n        max_n_float_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        value MaxFloats\\n\\\n    ) (\\n\\\n        sfunc = max_n_float_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = max_n_float_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = max_n_float_serialize,\\n\\\n        deserialfunc = max_n_float_deserialize,\\n\\\n        finalfunc = max_n_float_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_float_rollup\",\n    requires = [\n        max_n_float_rollup_trans,\n        max_n_float_final,\n        max_n_float_combine,\n        max_n_float_serialize,\n        max_n_float_deserialize\n    ],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn max_float_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val DOUBLE PRECISION, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}.0/128, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_array\n            let result = client\n                .update(\"SELECT into_array(max_n(val, 5)) from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<f64>>()\n                .unwrap();\n            assert_eq!(\n                result.unwrap(),\n                vec![99. / 128., 98. / 128., 97. / 128., 96. / 128., 95. / 128.]\n            );\n            let result = client\n                .update(\"SELECT max_n(val, 5)->into_array() from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<f64>>()\n                .unwrap();\n            assert_eq!(\n                result.unwrap(),\n                vec![99. / 128., 98. / 128., 97. / 128., 96. / 128., 95. / 128.]\n            );\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(max_n(val, 3))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"0.7734375\")\n            );\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0.765625\"));\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"0.7578125\")\n            );\n            assert!(result.next().is_none());\n            let mut result = client\n                .update(\n                    \"SELECT (max_n(val, 3)->into_values())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"0.7734375\")\n            );\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0.765625\"));\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"0.7578125\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let result =\n                client.update(\n                    \"WITH aggs as (SELECT category, max_n(val, 5) as agg from data GROUP BY category)\n                        SELECT into_array(rollup(agg)) FROM aggs\",\n                        None, &[],\n                    ).unwrap().first().get_one::<Vec<f64>>().unwrap();\n            assert_eq!(\n                result.unwrap(),\n                vec![99. / 128., 98. / 128., 97. / 128., 96. / 128., 95. / 128.]\n            );\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/max_int.rs",
    "content": "use pgrx::{iter::SetOfIterator, *};\n\nuse crate::nmost::*;\n\nuse crate::{\n    accessors::{AccessorIntoArray, AccessorIntoValues},\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\nuse std::cmp::Reverse;\n\ntype MaxIntTransType = NMostTransState<Reverse<i64>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MaxInts<'input> {\n        capacity : u32,\n        elements : u32,\n        values : [i64; self.elements],\n    }\n}\nron_inout_funcs!(MaxInts<'input>);\n\nimpl<'input> From<&mut MaxIntTransType> for MaxInts<'input> {\n    fn from(item: &mut MaxIntTransType) -> Self {\n        let heap = std::mem::take(&mut item.heap);\n        unsafe {\n            flatten!(MaxInts {\n                capacity: item.capacity as u32,\n                elements: heap.len() as u32,\n                values: heap\n                    .into_sorted_vec()\n                    .into_iter()\n                    .map(|x| x.0)\n                    .collect::<Vec<i64>>()\n                    .into()\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_int_trans(\n    state: Internal,\n    value: i64,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_function(\n        unsafe { state.to_inner::<MaxIntTransType>() },\n        Reverse(value),\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_int_rollup_trans(\n    state: Internal,\n    value: MaxInts<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<Reverse<i64>> = value.values.clone().into_iter().map(Reverse).collect();\n    nmost_rollup_trans_function(\n        unsafe { state.to_inner::<MaxIntTransType>() },\n        &values,\n        value.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_int_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_combine(\n        unsafe { state1.to_inner::<MaxIntTransType>() },\n        unsafe { state2.to_inner::<MaxIntTransType>() },\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_int_serialize(state: Internal) -> bytea {\n    let state: Inner<MaxIntTransType> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_int_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: MaxIntTransType = crate::do_deserialize!(bytes, MaxIntTransType);\n    Internal::new(i).into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_int_final(state: Internal) -> MaxInts<'static> {\n    unsafe { &mut *state.to_inner::<MaxIntTransType>().unwrap() }.into()\n}\n\n#[pg_extern(name = \"into_array\", immutable, parallel_safe)]\npub fn max_n_int_to_array(agg: MaxInts<'static>) -> Vec<i64> {\n    agg.values.clone().into_vec()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn max_n_int_to_values(agg: MaxInts<'static>) -> SetOfIterator<'static, i64> {\n    SetOfIterator::new(agg.values.clone().into_iter())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_int_into_values(\n    agg: MaxInts<'static>,\n    _accessor: AccessorIntoValues,\n) -> SetOfIterator<'static, i64> {\n    max_n_int_to_values(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_int_into_array(agg: MaxInts<'static>, _accessor: AccessorIntoArray) -> Vec<i64> {\n    max_n_int_to_array(agg)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE max_n(\\n\\\n        value bigint, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = max_n_int_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = max_n_int_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = max_n_int_serialize,\\n\\\n        deserialfunc = max_n_int_deserialize,\\n\\\n        finalfunc = max_n_int_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_int\",\n    requires = [\n        max_n_int_trans,\n        max_n_int_final,\n        max_n_int_combine,\n        max_n_int_serialize,\n        max_n_int_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        value MaxInts\\n\\\n    ) (\\n\\\n        sfunc = max_n_int_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = max_n_int_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = max_n_int_serialize,\\n\\\n        deserialfunc = max_n_int_deserialize,\\n\\\n        finalfunc = max_n_int_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_int_rollup\",\n    requires = [\n        max_n_int_rollup_trans,\n        max_n_int_final,\n        max_n_int_combine,\n        max_n_int_serialize,\n        max_n_int_deserialize\n    ],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn max_int_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\"CREATE TABLE data(val INT8, category INT)\", None, &[])\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_array\n            let result = client\n                .update(\"SELECT into_array(max_n(val, 5)) from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<i64>>()\n                .unwrap();\n            assert_eq!(result.unwrap(), vec![99, 98, 97, 96, 95]);\n            let result = client\n                .update(\"SELECT max_n(val, 5)->into_array() from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<i64>>()\n                .unwrap();\n            assert_eq!(result.unwrap(), vec![99, 98, 97, 96, 95]);\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(max_n(val, 3))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"99\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"98\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"97\"));\n            assert!(result.next().is_none());\n            let mut result = client\n                .update(\n                    \"SELECT (max_n(val, 3)->into_values())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"99\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"98\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"97\"));\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let result =\n                client.update(\n                    \"WITH aggs as (SELECT category, max_n(val, 5) as agg from data GROUP BY category)\n                        SELECT into_array(rollup(agg)) FROM aggs\",\n                        None, &[],\n                    ).unwrap().first().get_one::<Vec<i64>>().unwrap();\n            assert_eq!(result.unwrap(), vec![99, 98, 97, 96, 95]);\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/max_time.rs",
    "content": "use pgrx::{iter::SetOfIterator, *};\n\nuse crate::nmost::*;\n\nuse crate::{\n    accessors::{AccessorIntoArray, AccessorIntoValues},\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\nuse std::cmp::Reverse;\n\ntype MaxTimeTransType = NMostTransState<Reverse<pg_sys::TimestampTz>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MaxTimes <'input> {\n        capacity : u32,\n        elements : u32,\n        values : [pg_sys::TimestampTz; self.elements],\n    }\n}\nron_inout_funcs!(MaxTimes<'input>);\n\nimpl<'input> From<&mut MaxTimeTransType> for MaxTimes<'input> {\n    fn from(item: &mut MaxTimeTransType) -> Self {\n        let heap = std::mem::take(&mut item.heap);\n        unsafe {\n            flatten!(MaxTimes {\n                capacity: item.capacity as u32,\n                elements: heap.len() as u32,\n                values: heap\n                    .into_sorted_vec()\n                    .into_iter()\n                    .map(|x| x.0)\n                    .collect::<Vec<pg_sys::TimestampTz>>()\n                    .into()\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_time_trans(\n    state: Internal,\n    value: crate::raw::TimestampTz,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_function(\n        unsafe { state.to_inner::<MaxTimeTransType>() },\n        Reverse(value.into()),\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_time_rollup_trans(\n    state: Internal,\n    value: MaxTimes<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<Reverse<pg_sys::TimestampTz>> =\n        value.values.clone().into_iter().map(Reverse).collect();\n    nmost_rollup_trans_function(\n        unsafe { state.to_inner::<MaxTimeTransType>() },\n        &values,\n        value.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_time_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_combine(\n        unsafe { state1.to_inner::<MaxTimeTransType>() },\n        unsafe { state2.to_inner::<MaxTimeTransType>() },\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_time_serialize(state: Internal) -> bytea {\n    let state: Inner<MaxTimeTransType> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_time_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: MaxTimeTransType = crate::do_deserialize!(bytes, MaxTimeTransType);\n    Internal::new(i).into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn max_n_time_final(state: Internal) -> MaxTimes<'static> {\n    unsafe { &mut *state.to_inner::<MaxTimeTransType>().unwrap() }.into()\n}\n\n#[pg_extern(name = \"into_array\", immutable, parallel_safe)]\npub fn max_n_time_to_array(agg: MaxTimes<'static>) -> Vec<crate::raw::TimestampTz> {\n    agg.values\n        .clone()\n        .into_iter()\n        .map(crate::raw::TimestampTz::from)\n        .collect()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn max_n_time_to_values(\n    agg: MaxTimes<'static>,\n) -> SetOfIterator<'static, crate::raw::TimestampTz> {\n    SetOfIterator::new(\n        agg.values\n            .clone()\n            .into_iter()\n            .map(crate::raw::TimestampTz::from),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_time_into_values(\n    agg: MaxTimes<'static>,\n    _accessor: AccessorIntoValues,\n) -> SetOfIterator<'static, crate::raw::TimestampTz> {\n    max_n_time_to_values(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_max_time_into_array(\n    agg: MaxTimes<'static>,\n    _accessor: AccessorIntoArray,\n) -> Vec<crate::raw::TimestampTz> {\n    max_n_time_to_array(agg)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE max_n(\\n\\\n        value timestamptz, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = max_n_time_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = max_n_time_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = max_n_time_serialize,\\n\\\n        deserialfunc = max_n_time_deserialize,\\n\\\n        finalfunc = max_n_time_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_time\",\n    requires = [\n        max_n_time_trans,\n        max_n_time_final,\n        max_n_time_combine,\n        max_n_time_serialize,\n        max_n_time_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        value MaxTimes\\n\\\n    ) (\\n\\\n        sfunc = max_n_time_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = max_n_time_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = max_n_time_serialize,\\n\\\n        deserialfunc = max_n_time_deserialize,\\n\\\n        finalfunc = max_n_time_final\\n\\\n    );\\n\\\n\",\n    name = \"max_n_time_rollup\",\n    requires = [\n        max_n_time_rollup_trans,\n        max_n_time_final,\n        max_n_time_combine,\n        max_n_time_serialize,\n        max_n_time_deserialize\n    ],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn max_time_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val TIMESTAMPTZ, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client.update(\n                    &format!(\"INSERT INTO data VALUES ('2020-1-1 UTC'::timestamptz + {} * '1d'::interval, {})\", i, i % 4),\n                    None,\n                    &[]\n                ).unwrap();\n            }\n\n            // Test into_array\n            let result = client\n                .update(\n                    \"SELECT into_array(max_n(val, 5))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<&str>()\n                .unwrap();\n            assert_eq!(result.unwrap(), \"{\\\"2020-04-09 00:00:00+00\\\",\\\"2020-04-08 00:00:00+00\\\",\\\"2020-04-07 00:00:00+00\\\",\\\"2020-04-06 00:00:00+00\\\",\\\"2020-04-05 00:00:00+00\\\"}\");\n            let result = client\n                .update(\n                    \"SELECT (max_n(val, 5)->into_array())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<&str>()\n                .unwrap();\n            assert_eq!(result.unwrap(), \"{\\\"2020-04-09 00:00:00+00\\\",\\\"2020-04-08 00:00:00+00\\\",\\\"2020-04-07 00:00:00+00\\\",\\\"2020-04-06 00:00:00+00\\\",\\\"2020-04-05 00:00:00+00\\\"}\");\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(max_n(val, 3))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-04-09 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-04-08 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-04-07 00:00:00+00\")\n            );\n            assert!(result.next().is_none());\n            let mut result = client\n                .update(\n                    \"SELECT (max_n(val, 3)->into_values())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-04-09 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-04-08 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-04-07 00:00:00+00\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let result =\n                client.update(\n                    \"WITH aggs as (SELECT category, max_n(val, 5) as agg from data GROUP BY category)\n                        SELECT into_array(rollup(agg))::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap().first().get_one::<&str>().unwrap();\n            assert_eq!(result.unwrap(), \"{\\\"2020-04-09 00:00:00+00\\\",\\\"2020-04-08 00:00:00+00\\\",\\\"2020-04-07 00:00:00+00\\\",\\\"2020-04-06 00:00:00+00\\\",\\\"2020-04-05 00:00:00+00\\\"}\");\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/min_by_float.rs",
    "content": "use pgrx::{iter::TableIterator, *};\n\nuse crate::nmost::min_float::*;\nuse crate::nmost::*;\n\nuse crate::{\n    build, flatten,\n    palloc::{Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\nuse ordered_float::NotNan;\n\ntype MinByFloatTransType = NMostByTransState<NotNan<f64>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MinByFloats<'input> {\n        values: MinFloatsData<'input>,  // Nesting pg_types adds 8 bytes of header\n        data: DatumStore<'input>,\n    }\n}\nron_inout_funcs!(MinByFloats<'input>);\n\nimpl<'input> From<MinByFloatTransType> for MinByFloats<'input> {\n    fn from(item: MinByFloatTransType) -> Self {\n        let (capacity, val_ary, data) = item.into_sorted_parts();\n        unsafe {\n            flatten!(MinByFloats {\n                values: build!(MinFloats {\n                    capacity: capacity as u32,\n                    elements: val_ary.len() as u32,\n                    values: val_ary\n                        .into_iter()\n                        .map(f64::from)\n                        .collect::<Vec<f64>>()\n                        .into()\n                })\n                .0,\n                data,\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_float_trans(\n    state: Internal,\n    value: f64,\n    data: AnyElement,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_trans_function(\n        unsafe { state.to_inner::<MinByFloatTransType>() },\n        NotNan::new(value).unwrap(),\n        data,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_float_rollup_trans(\n    state: Internal,\n    value: MinByFloats<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<NotNan<f64>> = value\n        .values\n        .values\n        .clone()\n        .into_iter()\n        .map(|x| NotNan::new(x).unwrap())\n        .collect();\n    nmost_by_rollup_trans_function(\n        unsafe { state.to_inner::<MinByFloatTransType>() },\n        &values,\n        &value.data,\n        value.values.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_float_final(state: Internal) -> MinByFloats<'static> {\n    unsafe { state.to_inner::<MinByFloatTransType>().unwrap().clone() }.into()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn min_n_by_float_to_values(\n    agg: MinByFloats<'static>,\n    _dummy: Option<AnyElement>,\n) -> TableIterator<'static, (name!(value, f64), name!(data, AnyElement))> {\n    TableIterator::new(\n        agg.values\n            .values\n            .clone()\n            .into_iter()\n            .zip(agg.data.clone().into_anyelement_iter()),\n    )\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE min_n_by(\\n\\\n        value double precision, data AnyElement, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = min_n_by_float_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = min_n_by_float_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_by_float\",\n    requires = [min_n_by_float_trans, min_n_by_float_final],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        MinByFloats\\n\\\n    ) (\\n\\\n        sfunc = min_n_by_float_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = min_n_by_float_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_by_float_rollup\",\n    requires = [min_n_by_float_rollup_trans, min_n_by_float_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn min_by_float_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val DOUBLE PRECISION, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}.0/128, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(min_n_by(val, data, 3), NULL::data)::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0,\\\"(0,0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.0078125,\\\"(0.0078125,1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.015625,\\\"(0.015625,2)\\\")\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let mut result =\n                client.update(\n                    \"WITH aggs as (SELECT category, min_n_by(val, data, 5) as agg from data GROUP BY category)\n                        SELECT into_values(rollup(agg), NULL::data)::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0,\\\"(0,0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.0078125,\\\"(0.0078125,1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.015625,\\\"(0.015625,2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.0234375,\\\"(0.0234375,3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0.03125,\\\"(0.03125,0)\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/min_by_int.rs",
    "content": "use pgrx::{iter::TableIterator, *};\n\nuse crate::nmost::min_int::*;\nuse crate::nmost::*;\n\nuse crate::{\n    build, flatten,\n    palloc::{Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\ntype MinByIntTransType = NMostByTransState<i64>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MinByInts<'input> {\n        values: MinIntsData<'input>,  // Nesting pg_types adds 8 bytes of header\n        data: DatumStore<'input>,\n    }\n}\nron_inout_funcs!(MinByInts<'input>);\n\nimpl<'input> From<MinByIntTransType> for MinByInts<'input> {\n    fn from(item: MinByIntTransType) -> Self {\n        let (capacity, val_ary, data) = item.into_sorted_parts();\n        unsafe {\n            flatten!(MinByInts {\n                values: build!(MinInts {\n                    capacity: capacity as u32,\n                    elements: val_ary.len() as u32,\n                    values: val_ary.into()\n                })\n                .0,\n                data,\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_int_trans(\n    state: Internal,\n    value: i64,\n    data: AnyElement,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_trans_function(\n        unsafe { state.to_inner::<MinByIntTransType>() },\n        value,\n        data,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_int_rollup_trans(\n    state: Internal,\n    value: MinByInts<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_rollup_trans_function(\n        unsafe { state.to_inner::<MinByIntTransType>() },\n        value.values.values.as_slice(),\n        &value.data,\n        value.values.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_int_final(state: Internal) -> MinByInts<'static> {\n    unsafe { state.to_inner::<MinByIntTransType>().unwrap().clone() }.into()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn min_n_by_int_to_values(\n    agg: MinByInts<'static>,\n    _dummy: Option<AnyElement>,\n) -> TableIterator<'static, (name!(value, i64), name!(data, AnyElement))> {\n    TableIterator::new(\n        agg.values\n            .values\n            .clone()\n            .into_iter()\n            .zip(agg.data.clone().into_anyelement_iter()),\n    )\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE min_n_by(\\n\\\n        value bigint, data AnyElement, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = min_n_by_int_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = min_n_by_int_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_by_int\",\n    requires = [min_n_by_int_trans, min_n_by_int_final],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        MinByInts\\n\\\n    ) (\\n\\\n        sfunc = min_n_by_int_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = min_n_by_int_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_by_int_rollup\",\n    requires = [min_n_by_int_rollup_trans, min_n_by_int_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn min_by_int_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\"CREATE TABLE data(val INT8, category INT)\", None, &[])\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(min_n_by(val, data, 3), NULL::data)::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0,\\\"(0,0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(1,\\\"(1,1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(2,\\\"(2,2)\\\")\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let mut result =\n                client.update(\n                    \"WITH aggs as (SELECT category, min_n_by(val, data, 5) as agg from data GROUP BY category)\n                        SELECT into_values(rollup(agg), NULL::data)::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(0,\\\"(0,0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(1,\\\"(1,1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(2,\\\"(2,2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(3,\\\"(3,3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(4,\\\"(4,0)\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/min_by_time.rs",
    "content": "use pgrx::{iter::TableIterator, *};\n\nuse crate::nmost::min_time::*;\nuse crate::nmost::*;\n\nuse crate::{\n    build, flatten,\n    palloc::{Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\ntype MinByTimeTransType = NMostByTransState<pg_sys::TimestampTz>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MinByTimes<'input> {\n        values: MinTimesData<'input>,  // Nesting pg_types adds 8 bytes of header\n        data: DatumStore<'input>,\n    }\n}\nron_inout_funcs!(MinByTimes<'input>);\n\nimpl<'input> From<MinByTimeTransType> for MinByTimes<'input> {\n    fn from(item: MinByTimeTransType) -> Self {\n        let (capacity, val_ary, data) = item.into_sorted_parts();\n        unsafe {\n            flatten!(MinByTimes {\n                values: build!(MinTimes {\n                    capacity: capacity as u32,\n                    elements: val_ary.len() as u32,\n                    values: val_ary.into()\n                })\n                .0,\n                data,\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_time_trans(\n    state: Internal,\n    value: crate::raw::TimestampTz,\n    data: AnyElement,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_trans_function(\n        unsafe { state.to_inner::<MinByTimeTransType>() },\n        value.into(),\n        data,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_time_rollup_trans(\n    state: Internal,\n    value: MinByTimes<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_by_rollup_trans_function(\n        unsafe { state.to_inner::<MinByTimeTransType>() },\n        value.values.values.as_slice(),\n        &value.data,\n        value.values.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_by_time_final(state: Internal) -> MinByTimes<'static> {\n    unsafe { state.to_inner::<MinByTimeTransType>().unwrap().clone() }.into()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn min_n_by_time_to_values(\n    agg: MinByTimes<'static>,\n    _dummy: Option<AnyElement>,\n) -> TableIterator<\n    'static,\n    (\n        name!(value, crate::raw::TimestampTz),\n        name!(data, AnyElement),\n    ),\n> {\n    TableIterator::new(\n        agg.values\n            .values\n            .clone()\n            .into_iter()\n            .map(crate::raw::TimestampTz::from)\n            .zip(agg.data.clone().into_anyelement_iter()),\n    )\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE min_n_by(\\n\\\n        value timestamptz, data AnyElement, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = min_n_by_time_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = min_n_by_time_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_by_time\",\n    requires = [min_n_by_time_trans, min_n_by_time_final],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        MinByTimes\\n\\\n    ) (\\n\\\n        sfunc = min_n_by_time_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = min_n_by_time_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_by_time_rollup\",\n    requires = [min_n_by_time_rollup_trans, min_n_by_time_final],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn min_by_time_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val TIMESTAMPTZ, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client.update(\n                    &format!(\"INSERT INTO data VALUES ('2020-1-1 UTC'::timestamptz + {} * '1d'::interval, {})\", i, i % 4),\n                    None,\n                    &[]\n                ).unwrap();\n            }\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(min_n_by(val, data, 3), NULL::data)::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-01 00:00:00+00\\\"\\\",0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-02 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-02 00:00:00+00\\\"\\\",1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-03 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-03 00:00:00+00\\\"\\\",2)\\\")\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let mut result =\n                client.update(\n                    \"WITH aggs as (SELECT category, min_n_by(val, data, 5) as agg from data GROUP BY category)\n                        SELECT into_values(rollup(agg), NULL::data)::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-01 00:00:00+00\\\"\\\",0)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-02 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-02 00:00:00+00\\\"\\\",1)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-03 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-03 00:00:00+00\\\"\\\",2)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-04 00:00:00+00\\\"\\\",3)\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",\\\"(\\\"\\\"2020-01-05 00:00:00+00\\\"\\\",0)\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/min_float.rs",
    "content": "use pgrx::{iter::SetOfIterator, *};\n\nuse crate::nmost::*;\n\nuse crate::{\n    accessors::{AccessorIntoArray, AccessorIntoValues},\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\nuse ordered_float::NotNan;\n\ntype MinFloatTransType = NMostTransState<NotNan<f64>>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MinFloats <'input> {\n        capacity : u32,\n        elements : u32,\n        values : [f64; self.elements],\n    }\n}\nron_inout_funcs!(MinFloats<'input>);\n\nimpl<'input> From<&mut MinFloatTransType> for MinFloats<'input> {\n    fn from(item: &mut MinFloatTransType) -> Self {\n        let heap = std::mem::take(&mut item.heap);\n        unsafe {\n            flatten!(MinFloats {\n                capacity: item.capacity as u32,\n                elements: heap.len() as u32,\n                values: heap\n                    .into_sorted_vec()\n                    .into_iter()\n                    .map(f64::from)\n                    .collect::<Vec<f64>>()\n                    .into()\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_float_trans(\n    state: Internal,\n    value: f64,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_function(\n        unsafe { state.to_inner::<MinFloatTransType>() },\n        NotNan::new(value).unwrap(),\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_float_rollup_trans(\n    state: Internal,\n    value: MinFloats<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    let values: Vec<NotNan<f64>> = value\n        .values\n        .clone()\n        .into_iter()\n        .map(|x| NotNan::new(x).unwrap())\n        .collect();\n    nmost_rollup_trans_function(\n        unsafe { state.to_inner::<MinFloatTransType>() },\n        &values,\n        value.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_float_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_combine(\n        unsafe { state1.to_inner::<MinFloatTransType>() },\n        unsafe { state2.to_inner::<MinFloatTransType>() },\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_float_serialize(state: Internal) -> bytea {\n    let state: Inner<MinFloatTransType> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_float_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: MinFloatTransType = crate::do_deserialize!(bytes, MinFloatTransType);\n    Internal::new(i).into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_float_final(state: Internal) -> MinFloats<'static> {\n    unsafe { &mut *state.to_inner::<MinFloatTransType>().unwrap() }.into()\n}\n\n#[pg_extern(name = \"into_array\", immutable, parallel_safe)]\npub fn min_n_float_to_array(agg: MinFloats<'static>) -> Vec<f64> {\n    agg.values.clone().into_vec()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn min_n_float_to_values(agg: MinFloats<'static>) -> SetOfIterator<'static, f64> {\n    SetOfIterator::new(agg.values.clone().into_iter())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_float_into_values(\n    agg: MinFloats<'static>,\n    _accessor: AccessorIntoValues,\n) -> SetOfIterator<'static, f64> {\n    min_n_float_to_values(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_float_into_array(\n    agg: MinFloats<'static>,\n    _accessor: AccessorIntoArray,\n) -> Vec<f64> {\n    min_n_float_to_array(agg)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE min_n(\\n\\\n        value double precision, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = min_n_float_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = min_n_float_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = min_n_float_serialize,\\n\\\n        deserialfunc = min_n_float_deserialize,\\n\\\n        finalfunc = min_n_float_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_float\",\n    requires = [\n        min_n_float_trans,\n        min_n_float_final,\n        min_n_float_combine,\n        min_n_float_serialize,\n        min_n_float_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        value MinFloats\\n\\\n    ) (\\n\\\n        sfunc = min_n_float_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = min_n_float_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = min_n_float_serialize,\\n\\\n        deserialfunc = min_n_float_deserialize,\\n\\\n        finalfunc = min_n_float_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_float_rollup\",\n    requires = [\n        min_n_float_rollup_trans,\n        min_n_float_final,\n        min_n_float_combine,\n        min_n_float_serialize,\n        min_n_float_deserialize\n    ],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn min_float_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val DOUBLE PRECISION, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}.0/128, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_array\n            let result = client\n                .update(\"SELECT into_array(min_n(val, 5)) from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<f64>>()\n                .unwrap();\n            assert_eq!(\n                result.unwrap(),\n                vec![0. / 128., 1. / 128., 2. / 128., 3. / 128., 4. / 128.]\n            );\n            let result = client\n                .update(\"SELECT min_n(val, 5)->into_array() from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<f64>>()\n                .unwrap();\n            assert_eq!(\n                result.unwrap(),\n                vec![0. / 128., 1. / 128., 2. / 128., 3. / 128., 4. / 128.]\n            );\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(min_n(val, 3))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0\"));\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"0.0078125\")\n            );\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0.015625\"));\n            assert!(result.next().is_none());\n            let mut result = client\n                .update(\n                    \"SELECT (min_n(val, 3)->into_values())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0\"));\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"0.0078125\")\n            );\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0.015625\"));\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let result =\n                client.update(\n                    \"WITH aggs as (SELECT category, min_n(val, 5) as agg from data GROUP BY category)\n                        SELECT into_array(rollup(agg)) FROM aggs\",\n                        None, &[],\n                    ).unwrap().first().get_one::<Vec<f64>>();\n            assert_eq!(\n                result.unwrap().unwrap(),\n                vec![0. / 128., 1. / 128., 2. / 128., 3. / 128., 4. / 128.]\n            );\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/min_int.rs",
    "content": "use pgrx::{iter::SetOfIterator, *};\n\nuse crate::nmost::*;\n\nuse crate::{\n    accessors::{AccessorIntoArray, AccessorIntoValues},\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\ntype MinIntTransType = NMostTransState<i64>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MinInts <'input> {\n        capacity : u32,\n        elements : u32,\n        values : [i64; self.elements],\n    }\n}\nron_inout_funcs!(MinInts<'input>);\n\nimpl<'input> From<&mut MinIntTransType> for MinInts<'input> {\n    fn from(item: &mut MinIntTransType) -> Self {\n        let heap = std::mem::take(&mut item.heap);\n        unsafe {\n            flatten!(MinInts {\n                capacity: item.capacity as u32,\n                elements: heap.len() as u32,\n                values: heap.into_sorted_vec().into()\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_int_trans(\n    state: Internal,\n    value: i64,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_function(\n        unsafe { state.to_inner::<MinIntTransType>() },\n        value,\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_int_rollup_trans(\n    state: Internal,\n    value: MinInts<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_rollup_trans_function(\n        unsafe { state.to_inner::<MinIntTransType>() },\n        value.values.as_slice(),\n        value.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_int_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_combine(\n        unsafe { state1.to_inner::<MinIntTransType>() },\n        unsafe { state2.to_inner::<MinIntTransType>() },\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_int_serialize(state: Internal) -> bytea {\n    let state: Inner<MinIntTransType> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_int_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: MinIntTransType = crate::do_deserialize!(bytes, MinIntTransType);\n    Internal::new(i).into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_int_final(state: Internal) -> MinInts<'static> {\n    unsafe { &mut *state.to_inner::<MinIntTransType>().unwrap() }.into()\n}\n\n#[pg_extern(name = \"into_array\", immutable, parallel_safe)]\npub fn min_n_int_to_array(agg: MinInts<'static>) -> Vec<i64> {\n    agg.values.clone().into_vec()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn min_n_int_to_values(agg: MinInts<'static>) -> SetOfIterator<'static, i64> {\n    SetOfIterator::new(agg.values.clone().into_iter())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_int_into_values(\n    agg: MinInts<'static>,\n    _accessor: AccessorIntoValues,\n) -> SetOfIterator<'static, i64> {\n    min_n_int_to_values(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_int_into_array(agg: MinInts<'static>, _accessor: AccessorIntoArray) -> Vec<i64> {\n    min_n_int_to_array(agg)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE min_n(\\n\\\n        value bigint, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = min_n_int_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = min_n_int_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = min_n_int_serialize,\\n\\\n        deserialfunc = min_n_int_deserialize,\\n\\\n        finalfunc = min_n_int_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_int\",\n    requires = [\n        min_n_int_trans,\n        min_n_int_final,\n        min_n_int_combine,\n        min_n_int_serialize,\n        min_n_int_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        value MinInts\\n\\\n    ) (\\n\\\n        sfunc = min_n_int_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = min_n_int_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = min_n_int_serialize,\\n\\\n        deserialfunc = min_n_int_deserialize,\\n\\\n        finalfunc = min_n_int_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_int_rollup\",\n    requires = [\n        min_n_int_rollup_trans,\n        min_n_int_final,\n        min_n_int_combine,\n        min_n_int_serialize,\n        min_n_int_deserialize\n    ],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn min_int_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\"CREATE TABLE data(val INT8, category INT)\", None, &[])\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client\n                    .update(\n                        &format!(\"INSERT INTO data VALUES ({}, {})\", i, i % 4),\n                        None,\n                        &[],\n                    )\n                    .unwrap();\n            }\n\n            // Test into_array\n            let result = client\n                .update(\"SELECT into_array(min_n(val, 5)) from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<i64>>()\n                .unwrap();\n            assert_eq!(result.unwrap(), vec![0, 1, 2, 3, 4]);\n            let result = client\n                .update(\"SELECT min_n(val, 5)->into_array() from data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<Vec<i64>>()\n                .unwrap();\n            assert_eq!(result.unwrap(), vec![0, 1, 2, 3, 4]);\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(min_n(val, 3))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"1\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"2\"));\n            assert!(result.next().is_none());\n            let mut result = client\n                .update(\n                    \"SELECT (min_n(val, 3)->into_values())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"0\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"1\"));\n            assert_eq!(result.next().unwrap()[1].value().unwrap(), Some(\"2\"));\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let result =\n                client.update(\n                    \"WITH aggs as (SELECT category, min_n(val, 5) as agg from data GROUP BY category)\n                        SELECT into_array(rollup(agg)) FROM aggs\",\n                        None, &[],\n                    ).unwrap().first().get_one::<Vec<i64>>().unwrap();\n            assert_eq!(result.unwrap(), vec![0, 1, 2, 3, 4]);\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost/min_time.rs",
    "content": "use pgrx::{iter::SetOfIterator, *};\n\nuse crate::nmost::*;\n\nuse crate::{\n    accessors::{AccessorIntoArray, AccessorIntoValues},\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n    raw::bytea,\n    ron_inout_funcs,\n};\n\ntype MinTimeTransType = NMostTransState<pg_sys::TimestampTz>;\n\npg_type! {\n    #[derive(Debug)]\n    struct MinTimes <'input> {\n        capacity : u32,\n        elements : u32,\n        values : [pg_sys::TimestampTz; self.elements],\n    }\n}\nron_inout_funcs!(MinTimes<'input>);\n\nimpl<'input> From<&mut MinTimeTransType> for MinTimes<'input> {\n    fn from(item: &mut MinTimeTransType) -> Self {\n        let heap = std::mem::take(&mut item.heap);\n        unsafe {\n            flatten!(MinTimes {\n                capacity: item.capacity as u32,\n                elements: heap.len() as u32,\n                values: heap.into_sorted_vec().into()\n            })\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_time_trans(\n    state: Internal,\n    value: crate::raw::TimestampTz,\n    capacity: i64,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_function(\n        unsafe { state.to_inner::<MinTimeTransType>() },\n        value.into(),\n        capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_time_rollup_trans(\n    state: Internal,\n    value: MinTimes<'static>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_rollup_trans_function(\n        unsafe { state.to_inner::<MinTimeTransType>() },\n        value.values.as_slice(),\n        value.capacity as usize,\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_time_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    nmost_trans_combine(\n        unsafe { state1.to_inner::<MinTimeTransType>() },\n        unsafe { state2.to_inner::<MinTimeTransType>() },\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_time_serialize(state: Internal) -> bytea {\n    let state: Inner<MinTimeTransType> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_time_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: MinTimeTransType = crate::do_deserialize!(bytes, MinTimeTransType);\n    Internal::new(i).into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn min_n_time_final(state: Internal) -> MinTimes<'static> {\n    unsafe { &mut *state.to_inner::<MinTimeTransType>().unwrap() }.into()\n}\n\n#[pg_extern(name = \"into_array\", immutable, parallel_safe)]\npub fn min_n_time_to_array(agg: MinTimes<'static>) -> Vec<crate::raw::TimestampTz> {\n    agg.values\n        .clone()\n        .into_iter()\n        .map(crate::raw::TimestampTz::from)\n        .collect()\n}\n\n#[pg_extern(name = \"into_values\", immutable, parallel_safe)]\npub fn min_n_time_to_values(\n    agg: MinTimes<'static>,\n) -> SetOfIterator<'static, crate::raw::TimestampTz> {\n    SetOfIterator::new(\n        agg.values\n            .clone()\n            .into_iter()\n            .map(crate::raw::TimestampTz::from),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_time_into_values(\n    agg: MinTimes<'static>,\n    _accessor: AccessorIntoValues,\n) -> SetOfIterator<'static, crate::raw::TimestampTz> {\n    min_n_time_to_values(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_min_time_into_array(\n    agg: MinTimes<'static>,\n    _accessor: AccessorIntoArray,\n) -> Vec<crate::raw::TimestampTz> {\n    min_n_time_to_array(agg)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE min_n(\\n\\\n        value timestamptz, capacity bigint\\n\\\n    ) (\\n\\\n        sfunc = min_n_time_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = min_n_time_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = min_n_time_serialize,\\n\\\n        deserialfunc = min_n_time_deserialize,\\n\\\n        finalfunc = min_n_time_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_time\",\n    requires = [\n        min_n_time_trans,\n        min_n_time_final,\n        min_n_time_combine,\n        min_n_time_serialize,\n        min_n_time_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        value MinTimes\\n\\\n    ) (\\n\\\n        sfunc = min_n_time_rollup_trans,\\n\\\n        stype = internal,\\n\\\n        combinefunc = min_n_time_combine,\\n\\\n        parallel = safe,\\n\\\n        serialfunc = min_n_time_serialize,\\n\\\n        deserialfunc = min_n_time_deserialize,\\n\\\n        finalfunc = min_n_time_final\\n\\\n    );\\n\\\n\",\n    name = \"min_n_time_rollup\",\n    requires = [\n        min_n_time_rollup_trans,\n        min_n_time_final,\n        min_n_time_combine,\n        min_n_time_serialize,\n        min_n_time_deserialize\n    ],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn min_time_correctness() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(val TIMESTAMPTZ, category INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for i in 0..100 {\n                let i = (i * 83) % 100; // mess with the ordering just a little\n\n                client.update(\n                    &format!(\"INSERT INTO data VALUES ('2020-1-1 UTC'::timestamptz + {} * '1d'::interval, {})\", i, i % 4),\n                    None,\n                    &[]\n                ).unwrap();\n            }\n\n            // Test into_array\n            let result = client\n                .update(\n                    \"SELECT into_array(min_n(val, 5))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<&str>()\n                .unwrap();\n            assert_eq!(result.unwrap(), \"{\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-02 00:00:00+00\\\",\\\"2020-01-03 00:00:00+00\\\",\\\"2020-01-04 00:00:00+00\\\",\\\"2020-01-05 00:00:00+00\\\"}\");\n            let result = client\n                .update(\n                    \"SELECT (min_n(val, 5)->into_array())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<&str>()\n                .unwrap();\n            assert_eq!(result.unwrap(), \"{\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-02 00:00:00+00\\\",\\\"2020-01-03 00:00:00+00\\\",\\\"2020-01-04 00:00:00+00\\\",\\\"2020-01-05 00:00:00+00\\\"}\");\n\n            // Test into_values\n            let mut result = client\n                .update(\n                    \"SELECT into_values(min_n(val, 3))::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-01-01 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-01-02 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-01-03 00:00:00+00\")\n            );\n            assert!(result.next().is_none());\n            let mut result = client\n                .update(\n                    \"SELECT (min_n(val, 3)->into_values())::TEXT from data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-01-01 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-01-02 00:00:00+00\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"2020-01-03 00:00:00+00\")\n            );\n            assert!(result.next().is_none());\n\n            // Test rollup\n            let result =\n                client.update(\n                    \"WITH aggs as (SELECT category, min_n(val, 5) as agg from data GROUP BY category)\n                        SELECT into_array(rollup(agg))::TEXT FROM aggs\",\n                        None, &[],\n                    ).unwrap().first().get_one::<&str>().unwrap();\n            assert_eq!(result.unwrap(), \"{\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-02 00:00:00+00\\\",\\\"2020-01-03 00:00:00+00\\\",\\\"2020-01-04 00:00:00+00\\\",\\\"2020-01-05 00:00:00+00\\\"}\");\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/nmost.rs",
    "content": "use pgrx::*;\n\nuse serde::{Deserialize, Serialize};\n\nuse crate::{\n    aggregate_utils::in_aggregate_context,\n    datum_utils::{deep_copy_datum, free_datum, DatumStore},\n    palloc::{Inner, Internal, InternalAsValue},\n};\n\nuse std::collections::BinaryHeap;\n\nmod max_float;\nmod max_int;\nmod max_time;\nmod min_float;\nmod min_int;\nmod min_time;\n\nmod max_by_float;\nmod max_by_int;\nmod max_by_time;\nmod min_by_float;\nmod min_by_int;\nmod min_by_time;\n\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct NMostTransState<T: Ord> {\n    capacity: usize,\n    heap: BinaryHeap<T>,\n}\n\nimpl<T: Ord> NMostTransState<T> {\n    fn new(capacity: usize, first_val: T) -> NMostTransState<T> {\n        let mut new_heap = NMostTransState {\n            capacity,\n            heap: BinaryHeap::with_capacity(capacity),\n        };\n\n        new_heap.new_entry(first_val);\n\n        new_heap\n    }\n\n    fn new_entry(&mut self, new_val: T) {\n        // If at capacity see if we need to replace something\n        if self.heap.len() == self.capacity {\n            if !self.belongs_in_heap(&new_val) {\n                return;\n            }\n\n            self.heap.pop();\n        }\n\n        self.heap.push(new_val)\n    }\n\n    fn belongs_in_heap(&self, val: &T) -> bool {\n        // Note that this will actually be '>' if T is a Reverse<...> type\n        val < self.heap.peek().unwrap()\n    }\n}\n\nimpl<T: Ord + Copy> From<(&[T], usize)> for NMostTransState<T> {\n    fn from(input: (&[T], usize)) -> Self {\n        let (vals, capacity) = input;\n        let mut state = Self::new(capacity, vals[0]);\n        for val in vals[1..].iter() {\n            state.new_entry(*val);\n        }\n        state\n    }\n}\n\nfn nmost_trans_function<T: Ord>(\n    state: Option<Inner<NMostTransState<T>>>,\n    val: T,\n    capacity: usize,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<NMostTransState<T>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            if state.is_none() {\n                return Internal::new(NMostTransState::<T>::new(capacity, val)).to_inner();\n            }\n\n            let mut state = state.unwrap();\n            state.new_entry(val);\n            Some(state)\n        })\n    }\n}\n\nfn nmost_rollup_trans_function<T: Ord + Copy>(\n    state: Option<Inner<NMostTransState<T>>>,\n    sorted_vals: &[T],\n    capacity: usize,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<NMostTransState<T>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            if let Some(mut state) = state {\n                for val in sorted_vals {\n                    // The values are sorted, so as soon as we find one that shouldn't be added, we're done\n                    if !state.belongs_in_heap(val) {\n                        return Some(state);\n                    }\n                    state.new_entry(*val);\n                }\n\n                Some(state)\n            } else {\n                Internal::new::<NMostTransState<T>>((sorted_vals, capacity).into()).to_inner()\n            }\n        })\n    }\n}\n\nfn nmost_trans_combine<T: Clone + Ord + Copy>(\n    first: Option<Inner<NMostTransState<T>>>,\n    second: Option<Inner<NMostTransState<T>>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<NMostTransState<T>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (first, second) {\n                (None, None) => None,\n                (None, Some(only)) | (Some(only), None) => Internal::new(only.clone()).to_inner(),\n                (Some(a), Some(b)) => {\n                    let mut a = a.clone();\n                    // This could be made more efficient by iterating in the appropriate order with an early exit, but would requiring ordering the other heap\n                    for entry in b.heap.iter() {\n                        a.new_entry(*entry);\n                    }\n                    Internal::new(a).to_inner()\n                }\n            }\n        })\n    }\n}\n\n// TODO: serialize and deserialize will need to be implemented with Datum handling code\n#[derive(Clone, Debug)]\npub struct NMostByTransState<T: Ord> {\n    values: NMostTransState<(T, usize)>,\n    data: Vec<pg_sys::Datum>,\n    oid: pg_sys::Oid,\n}\n\nimpl<T: Clone + Ord> NMostByTransState<T> {\n    fn new(capacity: usize, first_val: T, first_element: pgrx::AnyElement) -> NMostByTransState<T> {\n        // first entry will always have index 0\n        let first_val = (first_val, 0);\n        NMostByTransState {\n            values: NMostTransState::new(capacity, first_val),\n            data: vec![unsafe { deep_copy_datum(first_element.datum(), first_element.oid()) }],\n            oid: first_element.oid(),\n        }\n    }\n\n    fn new_entry(&mut self, new_val: T, new_element: pgrx::AnyElement) {\n        assert!(new_element.oid() == self.oid);\n        if self.data.len() < self.values.capacity {\n            // Not yet full, easy case\n            self.values.new_entry((new_val, self.data.len()));\n            self.data\n                .push(unsafe { deep_copy_datum(new_element.datum(), new_element.oid()) });\n        } else if self\n            .values\n            .belongs_in_heap(&(new_val.clone(), self.data.len()))\n        {\n            // Full and value belongs in the heap (using len() for this check just keeps us from\n            // succeeding if we tie the max heap element)\n\n            let (_, index_to_replace) = *self\n                .values\n                .heap\n                .peek()\n                .expect(\"Can't be empty in this case\");\n            let old_datum = std::mem::replace(&mut self.data[index_to_replace], unsafe {\n                deep_copy_datum(new_element.datum(), new_element.oid())\n            });\n            unsafe { free_datum(old_datum, new_element.oid()) };\n            self.values.new_entry((new_val, index_to_replace));\n        }\n    }\n\n    // Sort the trans state and break it into a tuple of (capacity, values array, datum_store)\n    fn into_sorted_parts(self) -> (usize, Vec<T>, DatumStore<'static>) {\n        let values = self.values;\n        let heap = values.heap;\n        let (val_ary, idx_ary): (Vec<T>, Vec<usize>) = heap.into_sorted_vec().into_iter().unzip();\n\n        let mut mapped_data = vec![];\n        for i in idx_ary {\n            mapped_data.push(self.data[i]);\n        }\n\n        (\n            values.capacity,\n            val_ary,\n            DatumStore::from((self.oid, mapped_data)),\n        )\n    }\n}\n\nimpl<T: Ord + Copy> From<(&[T], &DatumStore<'_>, usize)> for NMostByTransState<T> {\n    fn from(in_tuple: (&[T], &DatumStore, usize)) -> Self {\n        let (vals, data, capacity) = in_tuple;\n        let mut elements = data.clone().into_anyelement_iter();\n        let mut state = Self::new(capacity, vals[0], elements.next().unwrap());\n        for val in vals[1..].iter() {\n            state.new_entry(*val, elements.next().unwrap());\n        }\n        state\n    }\n}\n\nfn nmost_by_trans_function<T: Ord + Clone>(\n    state: Option<Inner<NMostByTransState<T>>>,\n    val: T,\n    data: pgrx::AnyElement,\n    capacity: usize,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<NMostByTransState<T>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            if state.is_none() {\n                return Internal::new(NMostByTransState::<T>::new(capacity, val, data)).to_inner();\n            }\n\n            let mut state = state.unwrap();\n            state.new_entry(val, data);\n            Some(state)\n        })\n    }\n}\n\nfn nmost_by_rollup_trans_function<T: Ord + Copy>(\n    state: Option<Inner<NMostByTransState<T>>>,\n    sorted_vals: &[T],\n    datum_store: &DatumStore,\n    capacity: usize,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<NMostByTransState<T>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            if let Some(mut state) = state {\n                for (val, element) in sorted_vals\n                    .iter()\n                    .zip(datum_store.clone().into_anyelement_iter())\n                {\n                    // The values are sorted, so as soon as we find one that shouldn't be added, we're done\n                    if !state.values.belongs_in_heap(&(*val, state.values.capacity)) {\n                        return Some(state);\n                    }\n                    state.new_entry(*val, element);\n                }\n\n                Some(state)\n            } else {\n                Internal::new::<NMostByTransState<T>>((sorted_vals, datum_store, capacity).into())\n                    .to_inner()\n            }\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/palloc.rs",
    "content": "use std::{\n    alloc::{GlobalAlloc, Layout, System},\n    ops::{Deref, DerefMut},\n    ptr::NonNull,\n};\n\nuse pgrx::*;\n\npub unsafe fn in_memory_context<T, F: FnOnce() -> T>(mctx: pg_sys::MemoryContext, f: F) -> T {\n    let prev_ctx = pg_sys::CurrentMemoryContext;\n    pg_sys::CurrentMemoryContext = mctx;\n    let t = f();\n    pg_sys::CurrentMemoryContext = prev_ctx;\n    t\n}\n\npub use pgrx::Internal;\n\n/// Extension trait to translate postgres-understood `pgrx::Internal` type into\n/// the well-typed pointer type `Option<Inner<T>>`.\n///\n/// # Safety\n///\n/// This trait should only ever be implemented for `pgrx::Internal`\n/// There is an lifetime constraint on the returned pointer, though this is\n/// currently implicit.\npub unsafe trait InternalAsValue {\n    // unsafe fn value_or<T, F: FnOnce() -> T>(&mut self) -> &mut T;\n    unsafe fn to_inner<T>(self) -> Option<Inner<T>>;\n}\n\nunsafe impl InternalAsValue for Internal {\n    // unsafe fn value_or<T, F: FnOnce() -> T>(&mut self, f: F) -> &mut T {\n    //     if let Some(t) = self.get_mut() {\n    //         t\n    //     }\n\n    //     *self = Internal::new(f());\n    //     self.get_mut().unwrap()\n    // }\n\n    unsafe fn to_inner<T>(self) -> Option<Inner<T>> {\n        self.unwrap()\n            .map(|p| Inner(NonNull::new(p.cast_mut_ptr()).unwrap()))\n    }\n}\n\n/// Extension trait to turn the typed pointers `Inner<...>` and\n/// `Option<Inner<...>>` into the postgres-understood `pgrx::Internal` type.\n///\n/// # Safety\n/// The value input must live as long as postgres expects. TODO more info\npub unsafe trait ToInternal {\n    fn internal(self) -> Option<Internal>;\n}\n\npub struct Inner<T>(pub NonNull<T>);\n\nimpl<T> Deref for Inner<T> {\n    type Target = T;\n\n    fn deref(&self) -> &Self::Target {\n        unsafe { self.0.as_ref() }\n    }\n}\n\nimpl<T> DerefMut for Inner<T> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        unsafe { self.0.as_mut() }\n    }\n}\n\nunsafe impl<T> ToInternal for Option<Inner<T>> {\n    fn internal(self) -> Option<Internal> {\n        self.map(|p| Internal::from(Some(pg_sys::Datum::from(p.0.as_ptr()))))\n    }\n}\n\nunsafe impl<T> ToInternal for Inner<T> {\n    fn internal(self) -> Option<Internal> {\n        Some(Internal::from(Some(pg_sys::Datum::from(self.0.as_ptr()))))\n    }\n}\n\nimpl<T> From<T> for Inner<T> {\n    fn from(t: T) -> Self {\n        unsafe { Internal::new(t).to_inner().unwrap() }\n    }\n}\n\n// TODO these last two should probably be `unsafe`\nunsafe impl<T> ToInternal for *mut T {\n    fn internal(self) -> Option<Internal> {\n        Some(Internal::from(Some(pg_sys::Datum::from(self))))\n    }\n}\n\nunsafe impl<T> ToInternal for *const T {\n    fn internal(self) -> Option<Internal> {\n        Some(Internal::from(Some(pg_sys::Datum::from(self))))\n    }\n}\n\n// By default rust will `abort()` the process when the allocator returns NULL.\n// Since many systems can't reliably determine when an allocation will cause the\n// process to run out of memory, and just rely on the OOM killer cleaning up\n// afterwards, this is acceptable for many workloads. However, `abort()`-ing a\n// Postgres will restart the database, and since we often run Postgres on\n// systems which _can_ reliably return NULL on out-of-memory, we would like to\n// take advantage of this to cleanly shut down a single transaction when we fail\n// to allocate. Long-term the solution for this likely involves the `oom=panic`\n// flag[1], but at the time of writing the flag is not yet stable.\n//\n// This allocator implements a partial solution for turning out-of-memory into\n// transaction-rollback instead of process-abort. It is a thin shim over the\n// System allocator that `panic!()`s when the System allocator returns `NULL`.\n// In the event that still have enough remaining memory to serve the panic, this\n// will unwind the stack all the way to transaction-rollback. In the event we\n// don't even have enough memory to handle unwinding this will merely abort the\n// process with a panic-in-panic instead of a memory-allocation-failure. Under\n// the assumption that we're more likely to fail due to a few large allocations\n// rather than a very large number of small allocations, it seems likely that we\n// will have some memory remaining for unwinding, and that this will reduce the\n// likelihood of aborts.\n//\n// [1] `oom=panic` tracking issue: https://github.com/rust-lang/rust/issues/43596\nstruct PanickingAllocator;\n\n#[global_allocator]\nstatic ALLOCATOR: PanickingAllocator = PanickingAllocator;\n\nunsafe impl GlobalAlloc for PanickingAllocator {\n    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {\n        let p = System.alloc(layout);\n        if p.is_null() {\n            panic!(\"Out of memory\")\n        }\n        p\n    }\n\n    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {\n        System.dealloc(ptr, layout)\n    }\n\n    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {\n        let p = System.alloc_zeroed(layout);\n        if p.is_null() {\n            panic!(\"Out of memory\")\n        }\n        p\n    }\n\n    unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {\n        let p = System.realloc(ptr, layout, new_size);\n        if p.is_null() {\n            panic!(\"Out of memory\")\n        }\n        p\n    }\n}\n"
  },
  {
    "path": "extension/src/pg_any_element.rs",
    "content": "use std::{\n    collections::HashMap,\n    hash::{Hash, Hasher},\n    mem::size_of,\n};\n\nuse pgrx::*;\n\nuse pg_sys::{Datum, Oid};\n\nuse crate::datum_utils::{deep_copy_datum, DatumHashBuilder};\n\n// Unable to implement PartialEq for AnyElement, so creating a local copy\npub struct PgAnyElement {\n    datum: Datum,\n    typoid: Oid,\n}\n\nimpl PgAnyElement {\n    // pub fn from_datum_clone(datum : Datum, typoid : Oid) -> PgAnyElement {\n    //     PgAnyElement {\n    //         datum : unsafe{deep_copy_datum(datum, typoid)},\n    //         typoid\n    //     }\n    // }\n\n    pub fn deep_copy_datum(&self) -> Datum {\n        unsafe { deep_copy_datum(self.datum, self.typoid) }\n    }\n}\n\nimpl PartialEq for PgAnyElement {\n    #[allow(clippy::field_reassign_with_default)]\n    fn eq(&self, other: &Self) -> bool {\n        unsafe {\n            if self.typoid != other.typoid {\n                false\n            } else {\n                // TODO JOSH can we avoid the type cache lookup here\n                let typ = self.typoid;\n                let tentry = pg_sys::lookup_type_cache(typ, pg_sys::TYPECACHE_EQ_OPR_FINFO as _);\n\n                let flinfo = if (*tentry).eq_opr_finfo.fn_addr.is_some() {\n                    &(*tentry).eq_opr_finfo\n                } else {\n                    pgrx::error!(\"no equality function\");\n                };\n\n                let size = size_of::<pg_sys::FunctionCallInfoBaseData>()\n                    + size_of::<pg_sys::NullableDatum>() * 2;\n                let info = pg_sys::palloc0(size) as pg_sys::FunctionCallInfo;\n\n                (*info).flinfo = flinfo as *const pg_sys::FmgrInfo as *mut pg_sys::FmgrInfo;\n                (*info).context = std::ptr::null_mut();\n                (*info).resultinfo = std::ptr::null_mut();\n                (*info).fncollation = (*tentry).typcollation;\n                (*info).isnull = false;\n                (*info).nargs = 2;\n\n                (*info).args.as_mut_slice(2)[0] = pg_sys::NullableDatum {\n                    value: self.datum,\n                    isnull: false,\n                };\n                (*info).args.as_mut_slice(2)[1] = pg_sys::NullableDatum {\n                    value: other.datum,\n                    isnull: false,\n                };\n                (*(*info).flinfo).fn_addr.unwrap()(info) != Datum::from(0)\n            }\n        }\n    }\n}\n\nimpl Eq for PgAnyElement {}\n\nimpl Hash for PgAnyElement {\n    fn hash<H: Hasher>(&self, state: &mut H) {\n        self.datum.value().hash(state);\n    }\n}\n\nimpl From<(Datum, Oid)> for PgAnyElement {\n    fn from(other: (Datum, Oid)) -> Self {\n        let (datum, typoid) = other;\n        PgAnyElement { datum, typoid }\n    }\n}\n\nimpl From<AnyElement> for PgAnyElement {\n    fn from(other: AnyElement) -> Self {\n        PgAnyElement {\n            datum: other.datum(),\n            typoid: other.oid(),\n        }\n    }\n}\n\npub struct PgAnyElementHashMap<V>(pub(crate) HashMap<PgAnyElement, V, DatumHashBuilder>);\n\nimpl<V> PgAnyElementHashMap<V> {\n    pub fn new(typoid: Oid, collation: Option<Oid>) -> Self {\n        PgAnyElementHashMap(HashMap::with_hasher(unsafe {\n            DatumHashBuilder::from_type_id(typoid, collation)\n        }))\n    }\n\n    pub(crate) fn with_hasher(hasher: DatumHashBuilder) -> Self {\n        PgAnyElementHashMap(HashMap::with_hasher(hasher))\n    }\n\n    pub fn typoid(&self) -> Oid {\n        self.0.hasher().type_id\n    }\n\n    // Passthroughs\n    pub fn contains_key(&self, k: &PgAnyElement) -> bool {\n        self.0.contains_key(k)\n    }\n    pub fn get(&self, k: &PgAnyElement) -> Option<&V> {\n        self.0.get(k)\n    }\n    pub fn get_mut(&mut self, k: &PgAnyElement) -> Option<&mut V> {\n        self.0.get_mut(k)\n    }\n    pub(crate) fn hasher(&self) -> &DatumHashBuilder {\n        self.0.hasher()\n    }\n    pub fn insert(&mut self, k: PgAnyElement, v: V) -> Option<V> {\n        self.0.insert(k, v)\n    }\n    pub fn len(&self) -> usize {\n        self.0.len()\n    }\n    pub fn remove(&mut self, k: &PgAnyElement) -> Option<V> {\n        self.0.remove(k)\n    }\n}\n"
  },
  {
    "path": "extension/src/range.rs",
    "content": "use counter_agg::range::I64Range;\nuse pgrx::{extension_sql, pg_sys};\nuse serde::{Deserialize, Serialize};\nuse std::convert::TryInto;\nuse std::slice;\n\nuse flat_serialize_macro::flat_serialize;\n\n#[allow(non_camel_case_types)]\npub type tstzrange = *mut pg_sys::varlena;\n\n// Derived from Postgres' range_deserialize: https://github.com/postgres/postgres/blob/27e1f14563cf982f1f4d71e21ef247866662a052/src/backend/utils/adt/rangetypes.c#L1779\n// but we modify because we only allow specific types of ranges, namely [) inclusive on left and exclusive on right, as this makes a lot of logic simpler, and allows for a standard way to represent a range.\n#[allow(clippy::missing_safety_doc)]\npub unsafe fn get_range(range: tstzrange) -> Option<I64Range> {\n    let range_bytes = get_toasted_bytes(&*range);\n    let mut range_bytes = &range_bytes[8..]; // don't care about the Header and Oid\n    let flags = *range_bytes.last().unwrap();\n    let mut range = I64Range {\n        left: None,\n        right: None,\n    };\n    if flags & RANGE_EMPTY != 0 {\n        return None;\n    }\n    if range_has_lbound(flags) {\n        let bytes = range_bytes[..8].try_into().unwrap();\n        range_bytes = &range_bytes[8..];\n        let mut left = i64::from_ne_bytes(bytes);\n        if !lbound_inclusive(flags) {\n            left += 1;\n        }\n        range.left = Some(left);\n    }\n    if range_has_rbound(flags) {\n        let bytes = range_bytes[..8].try_into().unwrap();\n        let mut right = i64::from_ne_bytes(bytes);\n        if rbound_inclusive(flags) {\n            right += 1;\n        }\n        range.right = Some(right);\n    }\n    Some(range)\n}\n\nunsafe fn get_toasted_bytes(ptr: &pg_sys::varlena) -> &[u8] {\n    let mut ptr = pg_sys::pg_detoast_datum_packed(ptr as *const _ as *mut _);\n    if pgrx::varatt_is_1b(ptr) {\n        ptr = pg_sys::pg_detoast_datum_copy(ptr as *const _ as *mut _);\n    }\n    let data_len = pgrx::varsize_any(ptr);\n    slice::from_raw_parts(ptr as *mut u8, data_len)\n}\n\nconst RANGE_EMPTY: u8 = 0x01;\nconst RANGE_LB_INC: u8 = 0x02;\nconst RANGE_UB_INC: u8 = 0x04;\nconst RANGE_LB_INF: u8 = 0x08;\nconst RANGE_UB_INF: u8 = 0x10;\nconst RANGE_LB_NULL: u8 = 0x20; // should never be used, but why not.\nconst RANGE_UB_NULL: u8 = 0x40; // should never be used, but why not.\n\nfn range_has_lbound(flags: u8) -> bool {\n    flags & (RANGE_EMPTY | RANGE_LB_NULL | RANGE_LB_INF) == 0\n}\n\nfn lbound_inclusive(flags: u8) -> bool {\n    flags & RANGE_LB_INC != 0\n}\n\nfn range_has_rbound(flags: u8) -> bool {\n    (flags) & (RANGE_EMPTY | RANGE_UB_NULL | RANGE_UB_INF) == 0\n}\nfn rbound_inclusive(flags: u8) -> bool {\n    flags & RANGE_UB_INC != 0\n}\n\n// `Option<...>` is not suitable for disk storage. `Option<...>` does not have a\n// well-defined layout, for instance, an `Option<u32>` takes 8 bytes while an\n// `Option<NonZeroU32>` only takes up 4 bytes, despite them both representing\n// 32-bit numbers. Worse from our perspective is that the layouts of these are\n// not stable and they can change arbitrarily in the future, so an `Option<u32>`\n// as created by rust 1.50 may not have the same bytes as one created by rust\n// 1.51. `DiskOption<...>` is `repr(C, u64)` and thus does have a well-defined\n// layout: 8 bytes for the tag-bit determining if it's `None` or `Some` followed\n// by `size_of::<T>()` bytes in which the type can be stored.\n// Before stabilization we should probably change the layout to\n// ```\n// flat_serialize! {\n//     is_some: bool,\n//     value: [T; self.is_some as u8],\n// }\n// ```\nflat_serialize! {\n    #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]\n    struct I64RangeWrapper {\n        is_present: u8,\n        has_left: u8,\n        has_right: u8,\n        padding: [u8; 5],\n        left: i64 if self.is_present == 1 && self.has_left == 1,\n        right: i64 if self.is_present == 1 && self.has_right == 1,\n    }\n}\nimpl I64RangeWrapper {\n    pub fn to_i64range(&self) -> Option<I64Range> {\n        if self.is_present == 0 {\n            return None;\n        }\n        Some(I64Range {\n            left: self.left,\n            right: self.right,\n        })\n    }\n\n    pub fn from_i64range(b: Option<I64Range>) -> Self {\n        match b {\n            Some(range) => Self {\n                is_present: 1,\n                has_left: range.left.is_some().into(),\n                has_right: range.right.is_some().into(),\n                padding: [0; 5],\n                left: range.left,\n                right: range.right,\n            },\n            None => Self {\n                is_present: 0,\n                has_left: 0,\n                has_right: 0,\n                padding: [0; 5],\n                left: None,\n                right: None,\n            },\n        }\n    }\n}\n\n// this introduces a timescaledb dependency, but only kind of,\nextension_sql!(\"\\n\\\nCREATE FUNCTION toolkit_experimental.time_bucket_range( bucket_width interval, ts timestamptz)\\n\\\nRETURNS tstzrange as $$\\n\\\nSELECT tstzrange(time_bucket(bucket_width, ts), time_bucket(bucket_width, ts + bucket_width), '[)');\\n\\\n$$\\n\\\nLANGUAGE SQL IMMUTABLE PARALLEL SAFE;\\n\\\n\",\nname = \"time_bucket_range\",\n);\n"
  },
  {
    "path": "extension/src/raw.rs",
    "content": "#![allow(non_camel_case_types)]\n\nuse pgrx::*;\nuse pgrx_sql_entity_graph::metadata::{\n    ArgumentError, Returns, ReturnsError, SqlMapping, SqlTranslatable,\n};\n\nextension_sql!(\n    \"\\n\\\n        CREATE SCHEMA toolkit_experimental;\\n\\\n    \",\n    name = \"create_experimental_schema\",\n    creates = [\n        Type(bytea),\n        Type(text),\n        Type(TimestampTz),\n        Type(AnyElement),\n        Type(tstzrange),\n        Type(Interval),\n        Type(regproc)\n    ],\n    bootstrap,\n);\n\n// TODO temporary holdover types while we migrate from nominal types to actual\n\nmacro_rules! raw_type {\n    ($name:ident, $tyid: path, $arrayid: path) => {\n        impl FromDatum for $name {\n            unsafe fn from_polymorphic_datum(\n                datum: pg_sys::Datum,\n                is_null: bool,\n                _typoid: pg_sys::Oid,\n            ) -> Option<Self>\n            where\n                Self: Sized,\n            {\n                if is_null {\n                    return None;\n                }\n                Some(Self(datum))\n            }\n        }\n\n        impl IntoDatum for $name {\n            fn into_datum(self) -> Option<pg_sys::Datum> {\n                Some(self.0)\n            }\n            fn type_oid() -> pg_sys::Oid {\n                $tyid\n            }\n        }\n\n        impl From<pg_sys::Datum> for $name {\n            fn from(d: pg_sys::Datum) -> Self {\n                Self(d)\n            }\n        }\n\n        impl From<$name> for pg_sys::Datum {\n            fn from(v: $name) -> Self {\n                v.0\n            }\n        }\n\n        // SAFETY: all calls to raw_type! use type names that are valid SQL\n        unsafe impl SqlTranslatable for $name {\n            fn argument_sql() -> Result<SqlMapping, ArgumentError> {\n                Ok(SqlMapping::literal(stringify!($name)))\n            }\n            fn return_sql() -> Result<Returns, ReturnsError> {\n                Ok(Returns::One(SqlMapping::literal(stringify!($name))))\n            }\n        }\n\n        unsafe impl<'fcx> callconv::ArgAbi<'fcx> for $name {\n            unsafe fn unbox_arg_unchecked(arg: callconv::Arg<'_, 'fcx>) -> Self {\n                let index = arg.index();\n                unsafe {\n                    arg.unbox_arg_using_from_datum()\n                        .unwrap_or_else(|| panic!(\"argument {index} must not be null\"))\n                }\n            }\n\n            unsafe fn unbox_nullable_arg(arg: callconv::Arg<'_, 'fcx>) -> nullable::Nullable<Self> {\n                unsafe { arg.unbox_arg_using_from_datum().into() }\n            }\n        }\n    };\n}\n\n#[derive(Clone, Copy)]\npub struct bytea(pub pg_sys::Datum);\n\nraw_type!(bytea, pg_sys::BYTEAOID, pg_sys::BYTEAARRAYOID);\n\nunsafe impl pgrx::callconv::BoxRet for bytea {\n    unsafe fn box_into<'fcx>(\n        self,\n        fcinfo: &mut pgrx::callconv::FcInfo<'fcx>,\n    ) -> pgrx::datum::Datum<'fcx> {\n        unsafe { fcinfo.return_raw_datum(self.0) }\n    }\n}\n\n#[derive(Clone, Copy)]\npub struct text(pub pg_sys::Datum);\n\nraw_type!(text, pg_sys::TEXTOID, pg_sys::TEXTARRAYOID);\n\npub struct TimestampTz(pub pg_sys::Datum);\n\nraw_type!(\n    TimestampTz,\n    pg_sys::TIMESTAMPTZOID,\n    pg_sys::TIMESTAMPTZARRAYOID\n);\n\nunsafe impl pgrx::callconv::BoxRet for TimestampTz {\n    unsafe fn box_into<'fcx>(\n        self,\n        fcinfo: &mut pgrx::callconv::FcInfo<'fcx>,\n    ) -> pgrx::datum::Datum<'fcx> {\n        unsafe { fcinfo.return_raw_datum(self.0) }\n    }\n}\n\nimpl From<TimestampTz> for pg_sys::TimestampTz {\n    fn from(tstz: TimestampTz) -> Self {\n        tstz.0.value() as _\n    }\n}\n\nimpl From<pg_sys::TimestampTz> for TimestampTz {\n    fn from(ts: pg_sys::TimestampTz) -> Self {\n        Self(pg_sys::Datum::from(ts))\n    }\n}\n\npub struct AnyElement(pub pg_sys::Datum);\n\nraw_type!(AnyElement, pg_sys::ANYELEMENTOID, pg_sys::ANYARRAYOID);\n\npub struct tstzrange(pub pg_sys::Datum);\n\nraw_type!(tstzrange, pg_sys::TSTZRANGEOID, pg_sys::TSTZRANGEARRAYOID);\n\npub struct Interval(pub pg_sys::Datum);\n\nraw_type!(Interval, pg_sys::INTERVALOID, pg_sys::INTERVALARRAYOID);\n\nunsafe impl pgrx::callconv::BoxRet for Interval {\n    unsafe fn box_into<'fcx>(\n        self,\n        fcinfo: &mut pgrx::callconv::FcInfo<'fcx>,\n    ) -> pgrx::datum::Datum<'fcx> {\n        unsafe { fcinfo.return_raw_datum(self.0) }\n    }\n}\n\nimpl From<i64> for Interval {\n    fn from(interval: i64) -> Self {\n        let interval = pg_sys::Interval {\n            time: interval,\n            ..Default::default()\n        };\n        let interval = unsafe {\n            let ptr =\n                pg_sys::palloc(std::mem::size_of::<pg_sys::Interval>()) as *mut pg_sys::Interval;\n            *ptr = interval;\n            Interval(pg_sys::Datum::from(ptr))\n        };\n        // Now we have a valid Interval in at least one sense.  But we have the\n        // microseconds in the `time` field and `day` and `month` are both 0,\n        // which is legal.  However, directly converting one of these to TEXT\n        // comes out quite ugly if the number of microseconds is greater than 1 day:\n        //   8760:02:00\n        // Should be:\n        //   365 days 00:02:00\n        // How does postgresql do it?  It happens in src/backend/utils/adt/timestamp.c:timestamp_mi:\n        //  result->time = dt1 - dt2;\n        //  result = DatumGetIntervalP(DirectFunctionCall1(interval_justify_hours,\n        //                                                 IntervalPGetDatum(result)));\n        // So if we want the same behavior, we need to call interval_justify_hours too:\n        let function_args = vec![Some(pg_sys::Datum::from(interval))];\n        unsafe { pgrx::direct_function_call(pg_sys::interval_justify_hours, &function_args) }\n            .expect(\"interval_justify_hours does not return None\")\n    }\n}\n\npub struct regproc(pub pg_sys::Datum);\n\nraw_type!(regproc, pg_sys::REGPROCOID, pg_sys::REGPROCARRAYOID);\n"
  },
  {
    "path": "extension/src/saturation.rs",
    "content": "//! Saturating Math for Integers\n\nuse pgrx::*;\n\n/// Computes x+y, saturating at the numeric bounds instead of overflowing\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\nfn saturating_add(x: i32, y: i32) -> i32 {\n    x.saturating_add(y)\n}\n\n/// Computes x+y, saturating at 0 for the minimum bound instead of i32::MIN\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\nfn saturating_add_pos(x: i32, y: i32) -> i32 {\n    // check to see if abs of y is greater than the abs of x?\n    let result = x.saturating_add(y);\n    if result > 0 {\n        result\n    } else {\n        0\n    }\n}\n\n/// Computes x-y, saturating at the numeric bounds instead of overflowing.\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\nfn saturating_sub(x: i32, y: i32) -> i32 {\n    x.saturating_sub(y)\n}\n\n/// Computes x-y, saturating at 0 for the minimum bound instead of i32::MIN\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\nfn saturating_sub_pos(x: i32, y: i32) -> i32 {\n    if y > x {\n        0\n    } else {\n        x.saturating_sub(y)\n    }\n}\n\n/// Computes x*y, saturating at the numeric bounds instead of overflowing\n#[pg_extern(schema = \"toolkit_experimental\", immutable, parallel_safe)]\nfn saturating_mul(x: i32, y: i32) -> i32 {\n    x.saturating_mul(y)\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_add_max() {\n        assert_eq!(i32::MAX, saturating_add(i32::MAX, 100));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_add_min() {\n        assert_eq!(i32::MIN, saturating_add(i32::MIN, -100));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_add_pos() {\n        assert_eq!(0, saturating_add_pos(200, -350));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_sub_max() {\n        assert_eq!(i32::MAX, saturating_sub(i32::MAX, -10));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_sub_min() {\n        assert_eq!(i32::MIN, saturating_sub(i32::MIN, 10));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_sub_pos() {\n        assert_eq!(0, saturating_sub_pos(i32::MIN, 10));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_mul_max() {\n        assert_eq!(i32::MAX, saturating_mul(i32::MAX, 2));\n    }\n\n    #[pg_test]\n    #[allow(arithmetic_overflow)]\n    fn test_saturating_mul_min() {\n        assert_eq!(i32::MIN, saturating_mul(i32::MAX, -2));\n    }\n}\n"
  },
  {
    "path": "extension/src/serialization/collations.rs",
    "content": "use std::{\n    ffi::{CStr, CString},\n    mem::{align_of, size_of, MaybeUninit},\n    os::raw::c_char,\n    slice,\n};\n\nuse flat_serialize::{impl_flat_serializable, FlatSerializable, WrapErr};\n\nuse serde::{Deserialize, Serialize};\n\nuse once_cell::sync::Lazy;\n\nuse pg_sys::Oid;\nuse pgrx::*;\n\n// TODO short collation serializer?\n\n/// `PgCollationId` provides provides the ability to serialize and deserialize\n/// collation Oids as `(namespace, name)` pairs.\n#[derive(Debug, Clone, Copy)]\n#[repr(transparent)]\npub struct PgCollationId(pub Oid);\n\nimpl_flat_serializable!(PgCollationId);\n\nimpl PgCollationId {\n    pub fn is_invalid(&self) -> bool {\n        self.0 == pg_sys::InvalidOid\n    }\n\n    pub fn to_option_oid(self) -> Option<Oid> {\n        if self.is_invalid() {\n            None\n        } else {\n            Some(self.0)\n        }\n    }\n}\n\n#[allow(non_upper_case_globals)]\nconst Anum_pg_collation_oid: u32 = 1;\n// https://github.com/postgres/postgres/blob/e955bd4b6c2bcdbd253837f6cf4c7520b98e69d4/src/include/catalog/pg_collation.dat\n#[allow(deprecated)] // From::from is non-const\npub(crate) const DEFAULT_COLLATION_OID: Oid = unsafe { Oid::from_u32_unchecked(100) };\n\n#[allow(non_camel_case_types)]\n#[derive(Copy, Clone)]\n#[repr(C)]\nstruct FormData_pg_collation {\n    oid: pg_sys::Oid,\n    collname: pg_sys::NameData,\n    collnamespace: pg_sys::Oid,\n    collowner: pg_sys::Oid,\n    collprovider: c_char,\n    collisdeterministic: bool,\n    collencoding: i32,\n    collcollate: pg_sys::NameData,\n    collctype: pg_sys::NameData,\n}\n\n#[allow(non_camel_case_types)]\ntype Form_pg_collation = *mut FormData_pg_collation;\n\n#[allow(non_camel_case_types)]\n#[derive(Copy, Clone)]\n#[repr(C)]\nstruct FormData_pg_database {\n    oid: Oid,\n    datname: pg_sys::NameData,\n    datdba: Oid,\n    encoding: i32,\n    datcollate: pg_sys::NameData,\n    // TODO more fields I'm ignoring\n}\n\n#[allow(non_camel_case_types)]\ntype Form_pg_database = *mut FormData_pg_database;\n\nstatic DEFAULT_COLLATION_NAME: Lazy<CString> = Lazy::new(|| unsafe {\n    let tuple = pg_sys::SearchSysCache1(\n        pg_sys::SysCacheIdentifier::DATABASEOID as _,\n        pg_sys::Datum::from(pg_sys::MyDatabaseId),\n    );\n    if tuple.is_null() {\n        pgrx::error!(\"no database info\");\n    }\n\n    let database_tuple: Form_pg_database = get_struct(tuple);\n    let collation_name = (*database_tuple).datcollate.data.as_ptr();\n    let collation_name_len = CStr::from_ptr(collation_name).to_bytes().len();\n    let collation_name = pg_sys::pg_server_to_any(\n        collation_name,\n        collation_name_len as _,\n        pg_sys::pg_enc::PG_UTF8 as _,\n    );\n    let collation_name = CStr::from_ptr(collation_name);\n    pg_sys::ReleaseSysCache(tuple);\n    CString::from(collation_name)\n});\n\nimpl Serialize for PgCollationId {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        unsafe {\n            let mut layout: Option<(&str, &str)> = None;\n            if self.is_invalid() {\n                return layout.serialize(serializer);\n            }\n\n            let tuple = pg_sys::SearchSysCache1(\n                pg_sys::SysCacheIdentifier::COLLOID as _,\n                pg_sys::Datum::from(self.0),\n            );\n            if tuple.is_null() {\n                pgrx::error!(\"no collation info for oid {}\", self.0.to_u32());\n            }\n\n            let collation_tuple: Form_pg_collation = get_struct(tuple);\n\n            let namespace = pg_sys::get_namespace_name((*collation_tuple).collnamespace);\n            if namespace.is_null() {\n                pgrx::error!(\n                    \"invalid schema oid {}\",\n                    (*collation_tuple).collnamespace.to_u32()\n                );\n            }\n\n            let namespace_len = CStr::from_ptr(namespace).to_bytes().len();\n            let namespace = pg_sys::pg_server_to_any(\n                namespace,\n                namespace_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let namespace = CStr::from_ptr(namespace);\n            let namespace = namespace.to_str().unwrap();\n\n            // the 'default' collation isn't really a collation, and we need to\n            // look in pg_database to discover what real collation is\n            let collation_name = if self.0 == DEFAULT_COLLATION_OID {\n                &*DEFAULT_COLLATION_NAME\n            } else {\n                let collation_name = (*collation_tuple).collname.data.as_ptr();\n                let collation_name_len = CStr::from_ptr(collation_name).to_bytes().len();\n                let collation_name = pg_sys::pg_server_to_any(\n                    collation_name,\n                    collation_name_len as _,\n                    pg_sys::pg_enc::PG_UTF8 as _,\n                );\n                CStr::from_ptr(collation_name)\n            };\n            let collation_name = collation_name.to_str().unwrap();\n\n            let qualified_name: (&str, &str) = (namespace, collation_name);\n            layout = Some(qualified_name);\n            let res = layout.serialize(serializer);\n\n            pg_sys::ReleaseSysCache(tuple);\n            res\n        }\n    }\n}\n\nimpl<'de> Deserialize<'de> for PgCollationId {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        use serde::de::Error;\n\n        let collation = <Option<(&str, &str)>>::deserialize(deserializer)?;\n        let (namespace, name) = match collation {\n            None => return Ok(Self(pg_sys::Oid::INVALID)),\n            Some(qualified_name) => qualified_name,\n        };\n\n        let (namespace, name) = (\n            CString::new(namespace).unwrap(),\n            CString::new(name).unwrap(),\n        );\n        let (namespace_len, name_len) = (namespace.to_bytes().len(), name.to_bytes().len());\n        unsafe {\n            let namespace = pg_sys::pg_any_to_server(\n                namespace.as_ptr(),\n                namespace_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let namespace = CStr::from_ptr(namespace);\n\n            let name = pg_sys::pg_any_to_server(\n                name.as_ptr(),\n                name_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let name = CStr::from_ptr(name);\n\n            let namespace_id = pg_sys::LookupExplicitNamespace(namespace.as_ptr(), true);\n            if namespace_id == pg_sys::InvalidOid {\n                return Err(D::Error::custom(format!(\"invalid namespace {namespace:?}\")));\n            }\n\n            // COLLNAMEENCNSP is based on a triple `(collname, collencoding, collnamespace)`,\n            // however, `(collname, collnamespace)` is enough to uniquely determine\n            // a collation, though we need to check both collencoding = -1 and\n            // collencoding = DatabaseEncoding\n            // see:\n            //   https://www.postgresql.org/docs/13/catalog-pg-collation.html\n            //   https://github.com/postgres/postgres/blob/e955bd4b6c2bcdbd253837f6cf4c7520b98e69d4/src/backend/commands/collationcmds.c#L246\n\n            let mut collation_id = pg_sys::GetSysCacheOid(\n                pg_sys::SysCacheIdentifier::COLLNAMEENCNSP as _,\n                Anum_pg_collation_oid as _,\n                pg_sys::Datum::from(name.as_ptr()),\n                pg_sys::Datum::from(pg_sys::GetDatabaseEncoding()),\n                pg_sys::Datum::from(namespace_id),\n                pg_sys::Datum::from(0), //unused\n            );\n\n            if collation_id == pg_sys::InvalidOid {\n                collation_id = pg_sys::GetSysCacheOid(\n                    pg_sys::SysCacheIdentifier::COLLNAMEENCNSP as _,\n                    Anum_pg_collation_oid as _,\n                    pg_sys::Datum::from(name.as_ptr()),\n                    pg_sys::Datum::from((-1isize) as usize),\n                    pg_sys::Datum::from(namespace_id),\n                    pg_sys::Datum::from(0), //unused\n                );\n            }\n\n            if collation_id == pg_sys::InvalidOid {\n                // The default collation doesn't necessarily exist in the\n                // collations catalog, so check that specially\n                if name == &**DEFAULT_COLLATION_NAME {\n                    return Ok(PgCollationId(DEFAULT_COLLATION_OID));\n                }\n                return Err(D::Error::custom(format!(\n                    \"invalid collation {namespace:?}.{name:?}\"\n                )));\n            }\n\n            Ok(PgCollationId(collation_id))\n        }\n    }\n}\n\nunsafe fn get_struct<T>(tuple: pg_sys::HeapTuple) -> *mut T {\n    //((char *) ((TUP)->t_data) + (TUP)->t_data->t_hoff)\n    let t_data: *mut u8 = (*tuple).t_data.cast();\n    let t_hoff = (*(*tuple).t_data).t_hoff;\n    t_data.add(t_hoff as usize).cast()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n\n    use super::PgCollationId;\n    use pgrx::{pg_sys, pg_test};\n\n    #[allow(deprecated)] // no const version\n    const COLLATION_ID_950: PgCollationId =\n        PgCollationId(unsafe { pg_sys::Oid::from_u32_unchecked(950) });\n    #[allow(deprecated)] // no const version\n    const COLLATION_ID_951: PgCollationId =\n        PgCollationId(unsafe { pg_sys::Oid::from_u32_unchecked(951) });\n\n    // TODO is there a way we can test more of this without making it flaky?\n    #[pg_test]\n    fn test_pg_collation_id_serialize_default_collation_ron() {\n        let serialized = ron::to_string(&PgCollationId(\n            crate::serialization::collations::DEFAULT_COLLATION_OID,\n        ))\n        .unwrap();\n        let deserialized: PgCollationId = ron::from_str(&serialized).unwrap();\n        assert_ne!(deserialized.0, pg_sys::Oid::INVALID);\n        let serialized = ron::to_string(&PgCollationId(\n            crate::serialization::collations::DEFAULT_COLLATION_OID,\n        ))\n        .unwrap();\n        let deserialized2: PgCollationId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized2.0, deserialized.0);\n    }\n\n    #[pg_test]\n    fn test_pg_collation_id_serialize_c_collation() {\n        let serialized = bincode::serialize(&COLLATION_ID_950).unwrap();\n        assert_eq!(\n            serialized,\n            vec![\n                1, 10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111, 103, 1, 0, 0,\n                0, 0, 0, 0, 0, 67\n            ]\n        );\n        let deserialized: PgCollationId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, COLLATION_ID_950.0);\n    }\n\n    // TODO this test may be too flaky depending on what the default collation actually is\n    #[pg_test]\n    fn test_pg_collation_id_serialize_c_collation_ron() {\n        let serialized = ron::to_string(&COLLATION_ID_950).unwrap();\n        assert_eq!(&*serialized, \"Some((\\\"pg_catalog\\\",\\\"C\\\"))\",);\n        let deserialized: PgCollationId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, COLLATION_ID_950.0);\n    }\n\n    #[pg_test]\n    fn test_pg_collation_id_serialize_posix_collation() {\n        let serialized = bincode::serialize(&COLLATION_ID_951).unwrap();\n        assert_eq!(\n            serialized,\n            vec![\n                1, 10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111, 103, 5, 0, 0,\n                0, 0, 0, 0, 0, 80, 79, 83, 73, 88\n            ]\n        );\n        let deserialized: PgCollationId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, COLLATION_ID_951.0);\n    }\n\n    // TODO this test may be too flaky depending on what the default collation actually is\n    #[pg_test]\n    fn test_pg_collation_id_serialize_posix_collation_ron() {\n        let serialized = ron::to_string(&COLLATION_ID_951).unwrap();\n        assert_eq!(&*serialized, \"Some((\\\"pg_catalog\\\",\\\"POSIX\\\"))\",);\n        let deserialized: PgCollationId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, COLLATION_ID_951.0);\n    }\n}\n"
  },
  {
    "path": "extension/src/serialization/functions.rs",
    "content": "use std::{\n    ffi::{CStr, CString},\n    mem::{align_of, size_of, MaybeUninit},\n    os::raw::c_char,\n    slice,\n};\n\nuse flat_serialize::{impl_flat_serializable, FlatSerializable, WrapErr};\n\nuse serde::{Deserialize, Serialize};\n\nuse pg_sys::{Datum, Oid};\nuse pgrx::*;\n\n/// `PgProcId` provides provides the ability to serialize and deserialize\n/// regprocedures as `namespace.name(args)`\n#[derive(Debug, Clone, Copy)]\n#[repr(transparent)]\npub struct PgProcId(pub Oid);\n\nimpl_flat_serializable!(PgProcId);\n\n// FIXME upstream to pgrx\n// TODO use this or regprocedureout()?\nunsafe extern \"C\" {\n    pub fn format_procedure_qualified(procedure_oid: pg_sys::Oid) -> *const c_char;\n}\n\nimpl Serialize for PgProcId {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        unsafe {\n            let qualified_name = format_procedure_qualified(self.0);\n            let len = CStr::from_ptr(qualified_name).to_bytes().len();\n            let qualified_name =\n                pg_sys::pg_server_to_any(qualified_name, len as _, pg_sys::pg_enc::PG_UTF8 as _);\n            let qualified_name = CStr::from_ptr(qualified_name);\n            let qualified_name = qualified_name.to_str().unwrap();\n            qualified_name.serialize(serializer)\n        }\n    }\n}\n\nimpl<'de> Deserialize<'de> for PgProcId {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        // FIXME pgrx wraps all functions in rust wrappers, which makes them\n        //       uncallable with DirectFunctionCall(). Is there a way to\n        //       export both?\n        unsafe extern \"C-unwind\" {\n            #[allow(improper_ctypes)]\n            fn regprocedurein(fcinfo: pg_sys::FunctionCallInfo) -> Datum;\n        }\n        let qualified_name = <&str>::deserialize(deserializer)?;\n        let qualified_name = CString::new(qualified_name).unwrap();\n        let oid = unsafe {\n            pg_sys::DirectFunctionCall1Coll(\n                Some(regprocedurein),\n                pg_sys::InvalidOid,\n                pg_sys::Datum::from(qualified_name.as_ptr()),\n            )\n        };\n\n        Ok(Self(Oid::from(oid.value() as u32)))\n    }\n}\n"
  },
  {
    "path": "extension/src/serialization/types.rs",
    "content": "use std::{\n    ffi::{CStr, CString},\n    mem::{align_of, size_of, MaybeUninit},\n    slice,\n};\n\nuse flat_serialize::{impl_flat_serializable, FlatSerializable, WrapErr};\n\nuse serde::{Deserialize, Serialize};\n\nuse pg_sys::Oid;\nuse pgrx::*;\n\n/// Possibly a premature optimization, `ShortTypId` provides the ability to\n/// serialize and deserialize type Oids as `(namespace, name)` pairs, special\n/// casing a number of types with hardcoded Oids that we expect to be common so\n/// that these types can be stored more compactly if desired.\n#[derive(Debug, Clone, Copy)]\n#[repr(transparent)]\npub struct ShortTypeId(pub Oid);\n\nimpl_flat_serializable!(ShortTypeId);\n\nimpl From<Oid> for ShortTypeId {\n    fn from(id: Oid) -> Self {\n        Self(id)\n    }\n}\n\nimpl From<ShortTypeId> for Oid {\n    fn from(id: ShortTypeId) -> Self {\n        id.0\n    }\n}\n\nimpl Serialize for ShortTypeId {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        ShortTypIdSerializer::from_oid(self.0).serialize(serializer)\n    }\n}\n\nimpl<'de> Deserialize<'de> for ShortTypeId {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        let type_id = ShortTypIdSerializer::deserialize(deserializer)?;\n        Ok(Self(type_id.to_oid()))\n    }\n}\n\n#[derive(Debug, Serialize, Deserialize)]\n#[allow(clippy::upper_case_acronyms)]\nenum ShortTypIdSerializer {\n    BOOL,\n    BYTEA,\n    CHAR,\n    NAME,\n    INT8,\n    INT2,\n    INT2VECTOR,\n    INT4,\n    REGPROC,\n    TEXT,\n    JSON,\n    XML,\n    POINT,\n    FLOAT4,\n    FLOAT8,\n    MACADDR8,\n    VARCHAR,\n    DATE,\n    TIME,\n    TIMESTAMP,\n    TIMESTAMPTZ,\n    INTERVAL,\n    TIMETZ,\n    JSONB,\n    BOOLARRAY,\n    BYTEAARRAY,\n    CHARARRAY,\n    NAMEARRAY,\n    INT8ARRAY,\n    INT2ARRAY,\n    INT4ARRAY,\n    TEXTARRAY,\n    FLOAT4ARRAY,\n    FLOAT8ARRAY,\n    DATEARRAY,\n    TIMEARRAY,\n    TIMESTAMPARRAY,\n    TIMESTAMPTZARRAY,\n    INTERVALARRAY,\n    TIMETZARRAY,\n    NUMERICARRAY,\n    JSONBARRAY,\n    #[serde(rename = \"Type\")]\n    Other(PgTypId),\n}\n\nimpl ShortTypIdSerializer {\n    pub fn from_oid(oid: Oid) -> Self {\n        use ShortTypIdSerializer::*;\n        match oid {\n            pg_sys::BOOLOID => BOOL,\n            pg_sys::BYTEAOID => BYTEA,\n            pg_sys::CHAROID => CHAR,\n            pg_sys::NAMEOID => NAME,\n            pg_sys::INT8OID => INT8,\n            pg_sys::INT2OID => INT2,\n            pg_sys::INT2VECTOROID => INT2VECTOR,\n            pg_sys::INT4OID => INT4,\n            pg_sys::REGPROCOID => REGPROC,\n            pg_sys::TEXTOID => TEXT,\n            pg_sys::JSONOID => JSON,\n            pg_sys::XMLOID => XML,\n            pg_sys::POINTOID => POINT,\n            pg_sys::FLOAT4OID => FLOAT4,\n            pg_sys::FLOAT8OID => FLOAT8,\n            pg_sys::MACADDR8OID => MACADDR8,\n            pg_sys::VARCHAROID => VARCHAR,\n            pg_sys::DATEOID => DATE,\n            pg_sys::TIMEOID => TIME,\n            pg_sys::TIMESTAMPOID => TIMESTAMP,\n            pg_sys::TIMESTAMPTZOID => TIMESTAMPTZ,\n            pg_sys::INTERVALOID => INTERVAL,\n            pg_sys::TIMETZOID => TIMETZ,\n            pg_sys::JSONBOID => JSONB,\n            pg_sys::BOOLARRAYOID => BOOLARRAY,\n            pg_sys::BYTEAARRAYOID => BYTEAARRAY,\n            pg_sys::CHARARRAYOID => CHARARRAY,\n            pg_sys::NAMEARRAYOID => NAMEARRAY,\n            pg_sys::INT8ARRAYOID => INT8ARRAY,\n            pg_sys::INT2ARRAYOID => INT2ARRAY,\n            pg_sys::INT4ARRAYOID => INT4ARRAY,\n            pg_sys::TEXTARRAYOID => TEXTARRAY,\n            pg_sys::FLOAT4ARRAYOID => FLOAT4ARRAY,\n            pg_sys::FLOAT8ARRAYOID => FLOAT8ARRAY,\n            pg_sys::DATEARRAYOID => DATEARRAY,\n            pg_sys::TIMEARRAYOID => TIMEARRAY,\n            pg_sys::TIMESTAMPARRAYOID => TIMESTAMPARRAY,\n            pg_sys::TIMESTAMPTZARRAYOID => TIMESTAMPTZARRAY,\n            pg_sys::INTERVALARRAYOID => INTERVALARRAY,\n            pg_sys::TIMETZARRAYOID => TIMETZARRAY,\n            pg_sys::NUMERICARRAYOID => NUMERICARRAY,\n            pg_sys::JSONBARRAYOID => JSONBARRAY,\n            other => Other(PgTypId(other)),\n        }\n    }\n\n    pub fn to_oid(&self) -> Oid {\n        use ShortTypIdSerializer::*;\n        match self {\n            BOOL => pg_sys::BOOLOID,\n            BYTEA => pg_sys::BYTEAOID,\n            CHAR => pg_sys::CHAROID,\n            NAME => pg_sys::NAMEOID,\n            INT8 => pg_sys::INT8OID,\n            INT2 => pg_sys::INT2OID,\n            INT2VECTOR => pg_sys::INT2VECTOROID,\n            INT4 => pg_sys::INT4OID,\n            REGPROC => pg_sys::REGPROCOID,\n            TEXT => pg_sys::TEXTOID,\n            JSON => pg_sys::JSONOID,\n            XML => pg_sys::XMLOID,\n            POINT => pg_sys::POINTOID,\n            FLOAT4 => pg_sys::FLOAT4OID,\n            FLOAT8 => pg_sys::FLOAT8OID,\n            MACADDR8 => pg_sys::MACADDR8OID,\n            VARCHAR => pg_sys::VARCHAROID,\n            DATE => pg_sys::DATEOID,\n            TIME => pg_sys::TIMEOID,\n            TIMESTAMP => pg_sys::TIMESTAMPOID,\n            TIMESTAMPTZ => pg_sys::TIMESTAMPTZOID,\n            INTERVAL => pg_sys::INTERVALOID,\n            TIMETZ => pg_sys::TIMETZOID,\n            JSONB => pg_sys::JSONBOID,\n            BOOLARRAY => pg_sys::BOOLARRAYOID,\n            BYTEAARRAY => pg_sys::BYTEAARRAYOID,\n            CHARARRAY => pg_sys::CHARARRAYOID,\n            NAMEARRAY => pg_sys::NAMEARRAYOID,\n            INT8ARRAY => pg_sys::INT8ARRAYOID,\n            INT2ARRAY => pg_sys::INT2ARRAYOID,\n            INT4ARRAY => pg_sys::INT4ARRAYOID,\n            TEXTARRAY => pg_sys::TEXTARRAYOID,\n            FLOAT4ARRAY => pg_sys::FLOAT4ARRAYOID,\n            FLOAT8ARRAY => pg_sys::FLOAT8ARRAYOID,\n            DATEARRAY => pg_sys::DATEARRAYOID,\n            TIMEARRAY => pg_sys::TIMEARRAYOID,\n            TIMESTAMPARRAY => pg_sys::TIMESTAMPARRAYOID,\n            TIMESTAMPTZARRAY => pg_sys::TIMESTAMPTZARRAYOID,\n            INTERVALARRAY => pg_sys::INTERVALARRAYOID,\n            TIMETZARRAY => pg_sys::TIMETZARRAYOID,\n            NUMERICARRAY => pg_sys::NUMERICARRAYOID,\n            JSONBARRAY => pg_sys::JSONBARRAYOID,\n            Other(other) => other.0,\n        }\n    }\n}\n\n/// `PgTypId` provides provides the ability to serialize and deserialize type\n/// Oids as `(namespace, name)` pairs.\n#[derive(Debug)]\n#[repr(transparent)]\npub struct PgTypId(pub Oid);\n\nimpl Serialize for PgTypId {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where\n        S: serde::Serializer,\n    {\n        unsafe {\n            let tuple = pg_sys::SearchSysCache1(\n                pg_sys::SysCacheIdentifier::TYPEOID as _,\n                pg_sys::Datum::from(self.0),\n            );\n            if tuple.is_null() {\n                pgrx::error!(\"no type info for oid {}\", self.0.to_u32());\n            }\n\n            let type_tuple: pg_sys::Form_pg_type = get_struct(tuple);\n\n            let namespace = pg_sys::get_namespace_name((*type_tuple).typnamespace);\n            if namespace.is_null() {\n                pgrx::error!(\"invalid schema oid {}\", (*type_tuple).typnamespace.to_u32());\n            }\n\n            let namespace_len = CStr::from_ptr(namespace).to_bytes().len();\n            let namespace = pg_sys::pg_server_to_any(\n                namespace,\n                namespace_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let namespace = CStr::from_ptr(namespace);\n            let namespace = namespace.to_str().unwrap();\n\n            let type_name = (*type_tuple).typname.data.as_ptr();\n            let type_name_len = CStr::from_ptr(type_name).to_bytes().len();\n            let type_name = pg_sys::pg_server_to_any(\n                type_name,\n                type_name_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let type_name = CStr::from_ptr(type_name);\n            let type_name = type_name.to_str().unwrap();\n\n            let qualified_name: (&str, &str) = (namespace, type_name);\n            let res = qualified_name.serialize(serializer);\n            pg_sys::ReleaseSysCache(tuple);\n            res\n        }\n    }\n}\n\nimpl<'de> Deserialize<'de> for PgTypId {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::Deserializer<'de>,\n    {\n        use serde::de::Error;\n\n        let (namespace, name) = <(&str, &str)>::deserialize(deserializer)?;\n        let (namespace, name) = (\n            CString::new(namespace).unwrap(),\n            CString::new(name).unwrap(),\n        );\n        let (namespace_len, name_len) = (namespace.to_bytes().len(), name.to_bytes().len());\n        unsafe {\n            let namespace = pg_sys::pg_any_to_server(\n                namespace.as_ptr(),\n                namespace_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let namespace = CStr::from_ptr(namespace);\n\n            let name = pg_sys::pg_any_to_server(\n                name.as_ptr(),\n                name_len as _,\n                pg_sys::pg_enc::PG_UTF8 as _,\n            );\n            let name = CStr::from_ptr(name);\n\n            let namespace_id = pg_sys::LookupExplicitNamespace(namespace.as_ptr(), true);\n            if namespace_id == pg_sys::InvalidOid {\n                return Err(D::Error::custom(format!(\"invalid namespace {namespace:?}\")));\n            }\n\n            let type_id = pg_sys::GetSysCacheOid(\n                pg_sys::SysCacheIdentifier::TYPENAMENSP as _,\n                pg_sys::Anum_pg_type_oid as _,\n                pg_sys::Datum::from(name.as_ptr()),\n                pg_sys::Datum::from(namespace_id),\n                pg_sys::Datum::from(0), //unused\n                pg_sys::Datum::from(0), //unused\n            );\n            if type_id == pg_sys::InvalidOid {\n                return Err(D::Error::custom(format!(\n                    \"invalid type {namespace:?}.{name:?}\"\n                )));\n            }\n\n            Ok(PgTypId(type_id))\n        }\n    }\n}\n\nunsafe fn get_struct<T>(tuple: pg_sys::HeapTuple) -> *mut T {\n    //((char *) ((TUP)->t_data) + (TUP)->t_data->t_hoff)\n    let t_data: *mut u8 = (*tuple).t_data.cast();\n    let t_hoff = (*(*tuple).t_data).t_hoff;\n    t_data.add(t_hoff as usize).cast()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n\n    use super::{PgTypId, ShortTypeId};\n    use pgrx::{\n        pg_sys::{BOOLOID, CHAROID, CIRCLEOID},\n        pg_test,\n    };\n\n    #[pg_test]\n    fn test_pg_type_id_serialize_char_type() {\n        let serialized = bincode::serialize(&PgTypId(CHAROID)).unwrap();\n        assert_eq!(\n            serialized,\n            vec![\n                10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111, 103, 4, 0, 0, 0,\n                0, 0, 0, 0, 99, 104, 97, 114\n            ]\n        );\n        let deserialized: PgTypId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, CHAROID);\n    }\n\n    #[pg_test]\n    fn test_pg_type_id_serialize_char_type_ron() {\n        let serialized = ron::to_string(&PgTypId(CHAROID)).unwrap();\n        assert_eq!(&*serialized, \"(\\\"pg_catalog\\\",\\\"char\\\")\",);\n        let deserialized: PgTypId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, CHAROID);\n    }\n\n    #[pg_test]\n    fn test_pg_type_id_serialize_bool_type() {\n        let serialized = bincode::serialize(&PgTypId(BOOLOID)).unwrap();\n        assert_eq!(\n            serialized,\n            vec![\n                10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111, 103, 4, 0, 0, 0,\n                0, 0, 0, 0, 98, 111, 111, 108\n            ]\n        );\n        let deserialized: PgTypId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, BOOLOID);\n    }\n    #[pg_test]\n    fn test_pg_type_id_serialize_bool_type_ron() {\n        let serialized = ron::to_string(&PgTypId(BOOLOID)).unwrap();\n        assert_eq!(&*serialized, \"(\\\"pg_catalog\\\",\\\"bool\\\")\",);\n        let deserialized: PgTypId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, BOOLOID);\n    }\n\n    #[pg_test]\n    fn test_short_type_id_serialize_char_type() {\n        let serialized = bincode::serialize(&ShortTypeId(CHAROID)).unwrap();\n        assert_eq!(serialized, vec![2, 0, 0, 0],);\n        let deserialized: ShortTypeId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, CHAROID);\n    }\n\n    #[pg_test]\n    fn test_short_type_id_serialize_char_type_ron() {\n        let serialized = ron::to_string(&ShortTypeId(CHAROID)).unwrap();\n        assert_eq!(&*serialized, \"CHAR\",);\n        let deserialized: ShortTypeId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, CHAROID);\n    }\n\n    #[pg_test]\n    fn test_short_type_id_serialize_bool_type() {\n        let serialized = bincode::serialize(&ShortTypeId(BOOLOID)).unwrap();\n        assert_eq!(serialized, vec![0, 0, 0, 0],);\n        let deserialized: ShortTypeId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, BOOLOID);\n    }\n\n    #[pg_test]\n    fn test_short_type_id_serialize_bool_type_ron() {\n        let serialized = ron::to_string(&ShortTypeId(BOOLOID)).unwrap();\n        assert_eq!(&*serialized, \"BOOL\",);\n        let deserialized: ShortTypeId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, BOOLOID);\n    }\n\n    #[pg_test]\n    fn test_short_type_id_serialize_circle_type() {\n        let serialized = bincode::serialize(&ShortTypeId(CIRCLEOID)).unwrap();\n        assert_eq!(\n            serialized,\n            vec![\n                42, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 112, 103, 95, 99, 97, 116, 97, 108, 111, 103,\n                6, 0, 0, 0, 0, 0, 0, 0, 99, 105, 114, 99, 108, 101\n            ],\n        );\n        let deserialized: ShortTypeId = bincode::deserialize(&serialized).unwrap();\n        assert_eq!(deserialized.0, CIRCLEOID);\n    }\n\n    #[pg_test]\n    fn test_short_type_id_serialize_circle_type_ron() {\n        let serialized = ron::to_string(&ShortTypeId(CIRCLEOID)).unwrap();\n        assert_eq!(&*serialized, \"Type((\\\"pg_catalog\\\",\\\"circle\\\"))\");\n        let deserialized: ShortTypeId = ron::from_str(&serialized).unwrap();\n        assert_eq!(deserialized.0, CIRCLEOID);\n    }\n}\n"
  },
  {
    "path": "extension/src/serialization.rs",
    "content": "pub use self::collations::PgCollationId;\npub use self::functions::PgProcId;\npub use self::types::ShortTypeId;\nuse std::{\n    convert::TryInto,\n    os::raw::{c_char, c_int},\n};\n\nuse pgrx::pg_sys::{self};\n\nuse std::ffi::CStr;\n\npub(crate) mod collations;\nmod functions;\nmod types;\n\n// basically timestamptz_out\n#[unsafe(no_mangle)]\npub extern \"C\" fn _ts_toolkit_encode_timestamptz(\n    dt: pg_sys::TimestampTz,\n    buf: &mut [c_char; pg_sys::MAXDATELEN as _],\n) {\n    let mut tz: c_int = 0;\n    let mut tt: pg_sys::pg_tm = unsafe { std::mem::MaybeUninit::zeroed().assume_init() };\n    let mut fsec = 0;\n    let mut tzn = std::ptr::null();\n    unsafe {\n        if dt == pg_sys::TimestampTz::MAX || dt == pg_sys::TimestampTz::MIN {\n            return pg_sys::EncodeSpecialTimestamp(dt, buf.as_mut_ptr());\n        }\n        let err = pg_sys::timestamp2tm(\n            dt,\n            &mut tz,\n            &mut tt,\n            &mut fsec,\n            &mut tzn,\n            std::ptr::null_mut(),\n        );\n        if err != 0 {\n            panic!(\"timestamp out of range\")\n        }\n        pg_sys::EncodeDateTime(\n            &mut tt,\n            fsec,\n            true,\n            tz,\n            tzn,\n            pg_sys::DateStyle,\n            buf.as_mut_ptr(),\n        )\n    }\n}\n\n#[unsafe(no_mangle)]\n// this is only going to be used to communicate with a rust lib we compile with this one\n#[allow(improper_ctypes_definitions)]\npub extern \"C\" fn _ts_toolkit_decode_timestamptz(text: &str) -> i64 {\n    use std::{ffi::CString, mem::MaybeUninit, ptr};\n    let str = CString::new(text).unwrap();\n    unsafe {\n        let mut fsec = 0;\n        let mut tt = MaybeUninit::zeroed().assume_init();\n        let tm = &mut tt;\n        let mut tz = 0;\n        let mut dtype = 0;\n        let mut nf = 0;\n        let mut field = [ptr::null_mut(); pg_sys::MAXDATEFIELDS as _];\n        let mut ftype = [0; pg_sys::MAXDATEFIELDS as _];\n        let mut workbuf = [0; pg_sys::MAXDATELEN as usize + pg_sys::MAXDATEFIELDS as usize];\n        let mut dterr = pg_sys::ParseDateTime(\n            str.as_ptr(),\n            workbuf.as_mut_ptr(),\n            workbuf.len(),\n            field.as_mut_ptr(),\n            ftype.as_mut_ptr(),\n            pg_sys::MAXDATEFIELDS as i32,\n            &mut nf,\n        );\n        #[cfg(feature = \"pg15\")]\n        if dterr == 0 {\n            dterr = pg_sys::DecodeDateTime(\n                field.as_mut_ptr(),\n                ftype.as_mut_ptr(),\n                nf,\n                &mut dtype,\n                tm,\n                &mut fsec,\n                &mut tz,\n            )\n        }\n        #[cfg(not(feature = \"pg15\"))]\n        if dterr == 0 {\n            let mut extra = pgrx::pg_sys::DateTimeErrorExtra::default();\n            dterr = pg_sys::DecodeDateTime(\n                field.as_mut_ptr(),\n                ftype.as_mut_ptr(),\n                nf,\n                &mut dtype,\n                tm,\n                &mut fsec,\n                &mut tz,\n                &mut extra as *mut pgrx::pg_sys::DateTimeErrorExtra,\n            )\n        }\n\n        #[cfg(feature = \"pg15\")]\n        if dterr != 0 {\n            pg_sys::DateTimeParseError(\n                dterr,\n                str.as_ptr(),\n                c\"timestamptz\".as_ptr().cast::<c_char>(),\n            );\n        }\n        #[cfg(not(feature = \"pg15\"))]\n        if dterr != 0 {\n            pg_sys::DateTimeParseError(\n                dterr,\n                core::ptr::null_mut(),\n                str.as_ptr(),\n                c\"timestamptz\".as_ptr().cast::<c_char>(),\n                core::ptr::null_mut(),\n            );\n        }\n\n        match dtype as u32 {\n            pg_sys::DTK_DATE => {\n                let mut result = 0;\n                let err = pg_sys::tm2timestamp(tm, fsec, &mut tz, &mut result);\n                if err != 0 {\n                    // TODO pgrx error with correct errcode?\n                    panic!(\"timestamptz \\\"{text}\\\" out of range\")\n                }\n                result\n            }\n            pg_sys::DTK_EPOCH => pg_sys::SetEpochTimestamp(),\n            pg_sys::DTK_LATE => pg_sys::TimestampTz::MAX,\n            pg_sys::DTK_EARLY => pg_sys::TimestampTz::MIN,\n            _ => panic!(\"unexpected result {dtype} when parsing timestamptz \\\"{text}\\\"\"),\n        }\n    }\n}\n\npub enum EncodedStr<'s> {\n    Utf8(&'s str),\n    Other(&'s CStr),\n}\n\npub fn str_to_db_encoding(s: &str) -> EncodedStr<'_> {\n    if unsafe { pg_sys::GetDatabaseEncoding() == pg_sys::pg_enc::PG_UTF8 as i32 } {\n        return EncodedStr::Utf8(s);\n    }\n\n    let bytes = s.as_bytes();\n    let encoded = unsafe {\n        pg_sys::pg_any_to_server(\n            bytes.as_ptr() as *const c_char,\n            bytes.len().try_into().unwrap(),\n            pg_sys::pg_enc::PG_UTF8 as _,\n        )\n    };\n    if std::ptr::eq(encoded, bytes.as_ptr() as *const c_char) {\n        return EncodedStr::Utf8(s);\n    }\n\n    let cstr = unsafe { CStr::from_ptr(encoded) };\n    EncodedStr::Other(cstr)\n}\n\npub fn str_from_db_encoding(s: &CStr) -> &str {\n    if unsafe { pg_sys::GetDatabaseEncoding() == pg_sys::pg_enc::PG_UTF8 as i32 } {\n        return s.to_str().unwrap();\n    }\n\n    let str_len = s.to_bytes().len().try_into().unwrap();\n    let encoded =\n        unsafe { pg_sys::pg_server_to_any(s.as_ptr(), str_len, pg_sys::pg_enc::PG_UTF8 as _) };\n    if std::ptr::eq(encoded, s.as_ptr()) {\n        //TODO redundant check?\n        return s.to_str().unwrap();\n    }\n    unsafe { CStr::from_ptr(encoded).to_str().unwrap() }\n}\n\npub(crate) mod serde_reference_adaptor {\n    pub(crate) fn default_padding() -> [u8; 3] {\n        [0; 3]\n    }\n\n    pub(crate) fn default_header() -> u32 {\n        0\n    }\n}\n"
  },
  {
    "path": "extension/src/stabilization_info.rs",
    "content": "// This file serves as the canonical database for what functionality Toolkit has\n// stabilized and in which version they were stabilized. The file is consumed by\n// by post-install as well as a number of tests that check if our stabilization\n// guarantees are being upheld. These different usages require different views\n// of the same info, so to avoid parsing issues the stabilization data is\n// exposed as macros that're left to the other files to interpret.\n//\n// XXX this file is used as multiple modules. Search for `#[path = \"...\"]`\n//     directives before adding new macros to make sure that all relevant usages\n//     can handle it.\n\ncrate::functions_stabilized_at! {\n    STABLE_FUNCTIONS\n    \"1.20.0\" => {\n        total(tdigest),\n        total(uddsketch),\n    }\n    \"1.16.0\" => {\n        approx_count_distinct(anyelement),\n        approx_count_distinct_trans(internal,anyelement),\n        accessornumgaps_in(cstring),\n        accessornumgaps_out(accessornumgaps),\n        accessornumliveranges_in(cstring),\n        accessornumliveranges_out(accessornumliveranges),\n        arrow_heartbeat_agg_num_gaps(heartbeatagg,accessornumgaps),\n        arrow_heartbeat_agg_num_live_ranges(heartbeatagg,accessornumliveranges),\n        arrow_heartbeat_agg_trim_to(heartbeatagg,heartbeattrimtoaccessor),\n        heartbeattrimtoaccessor_in(cstring),\n        heartbeattrimtoaccessor_out(heartbeattrimtoaccessor),\n        num_gaps(),\n        num_gaps(heartbeatagg),\n        num_live_ranges(),\n        num_live_ranges(heartbeatagg),\n        trim_to(heartbeatagg,timestamp with time zone,interval),\n        trim_to(timestamp with time zone,interval),\n        accessorpercentilearray_in(cstring),\n        accessorpercentilearray_out(accessorpercentilearray),\n        arrow_uddsketch_approx_percentile_array(uddsketch,accessorpercentilearray),\n        days_in_month(timestamp with time zone),\n        month_normalize(double precision,timestamp with time zone,double precision),\n        to_epoch(timestamp with time zone),\n        accessorintoarray_in(cstring),\n        accessorintoarray_out(accessorintoarray),\n        arrow_max_float_into_array(maxfloats,accessorintoarray),\n        arrow_max_float_into_values(maxfloats,accessorintovalues),\n        into_array(),\n        into_array(maxfloats),\n        into_array(maxints),\n        into_array(maxtimes),\n        into_array(minfloats),\n        into_array(minints),\n        into_array(mintimes),\n        into_values(maxbyfloats,anyelement),\n        into_values(maxbyints,anyelement),\n        into_values(maxbytimes,anyelement),\n        into_values(maxfloats),\n        into_values(maxints),\n        into_values(maxtimes),\n        into_values(minbyfloats,anyelement),\n        into_values(minbyints,anyelement),\n        into_values(minbytimes,anyelement),\n        into_values(minfloats),\n        into_values(minints),\n        into_values(mintimes),\n        max_n(bigint,bigint),\n        max_n(double precision,bigint),\n        max_n(timestamp with time zone,bigint),\n        max_n_by(bigint,anyelement,bigint),\n        max_n_by(double precision,anyelement,bigint),\n        max_n_by(timestamp with time zone,anyelement,bigint),\n        max_n_by_float_final(internal),\n        max_n_by_float_rollup_trans(internal,maxbyfloats),\n        max_n_by_float_trans(internal,double precision,anyelement,bigint),\n        max_n_by_int_final(internal),\n        max_n_by_int_rollup_trans(internal,maxbyints),\n        max_n_by_int_trans(internal,bigint,anyelement,bigint),\n        max_n_by_time_final(internal),\n        max_n_by_time_rollup_trans(internal,maxbytimes),\n        max_n_by_time_trans(internal,timestamp with time zone,anyelement,bigint),\n        max_n_float_combine(internal,internal),\n        max_n_float_deserialize(bytea,internal),\n        max_n_float_final(internal),\n        max_n_float_rollup_trans(internal,maxfloats),\n        max_n_float_serialize(internal),\n        max_n_float_trans(internal,double precision,bigint),\n        max_n_int_combine(internal,internal),\n        max_n_int_deserialize(bytea,internal),\n        max_n_int_final(internal),\n        max_n_int_rollup_trans(internal,maxints),\n        max_n_int_serialize(internal),\n        max_n_int_trans(internal,bigint,bigint),\n        max_n_time_combine(internal,internal),\n        max_n_time_deserialize(bytea,internal),\n        max_n_time_final(internal),\n        max_n_time_rollup_trans(internal,maxtimes),\n        max_n_time_serialize(internal),\n        max_n_time_trans(internal,timestamp with time zone,bigint),\n        maxbyfloats_in(cstring),\n        maxbyfloats_out(maxbyfloats),\n        maxbyints_in(cstring),\n        maxbyints_out(maxbyints),\n        maxbytimes_in(cstring),\n        maxbytimes_out(maxbytimes),\n        maxfloats_in(cstring),\n        maxfloats_out(maxfloats),\n        maxints_in(cstring),\n        maxints_out(maxints),\n        maxtimes_in(cstring),\n        maxtimes_out(maxtimes),\n        min_n(bigint,bigint),\n        min_n(double precision,bigint),\n        min_n(timestamp with time zone,bigint),\n        min_n_by(bigint,anyelement,bigint),\n        min_n_by(double precision,anyelement,bigint),\n        min_n_by(timestamp with time zone,anyelement,bigint),\n        min_n_by_float_final(internal),\n        min_n_by_float_rollup_trans(internal,minbyfloats),\n        min_n_by_float_trans(internal,double precision,anyelement,bigint),\n        min_n_by_int_final(internal),\n        min_n_by_int_rollup_trans(internal,minbyints),\n        min_n_by_int_trans(internal,bigint,anyelement,bigint),\n        min_n_by_time_final(internal),\n        min_n_by_time_rollup_trans(internal,minbytimes),\n        min_n_by_time_trans(internal,timestamp with time zone,anyelement,bigint),\n        min_n_float_combine(internal,internal),\n        min_n_float_deserialize(bytea,internal),\n        min_n_float_final(internal),\n        min_n_float_rollup_trans(internal,minfloats),\n        min_n_float_serialize(internal),\n        min_n_float_trans(internal,double precision,bigint),\n        min_n_int_combine(internal,internal),\n        min_n_int_deserialize(bytea,internal),\n        min_n_int_final(internal),\n        min_n_int_rollup_trans(internal,minints),\n        min_n_int_serialize(internal),\n        min_n_int_trans(internal,bigint,bigint),\n        min_n_time_combine(internal,internal),\n        min_n_time_deserialize(bytea,internal),\n        min_n_time_final(internal),\n        min_n_time_rollup_trans(internal,mintimes),\n        min_n_time_serialize(internal),\n        min_n_time_trans(internal,timestamp with time zone,bigint),\n        minbyfloats_in(cstring),\n        minbyfloats_out(minbyfloats),\n        minbyints_in(cstring),\n        minbyints_out(minbyints),\n        minbytimes_in(cstring),\n        minbytimes_out(minbytimes),\n        minfloats_in(cstring),\n        minfloats_out(minfloats),\n        minints_in(cstring),\n        minints_out(minints),\n        mintimes_in(cstring),\n        mintimes_out(mintimes),\n        rollup(maxbyfloats),\n        rollup(maxbyints),\n        rollup(maxbytimes),\n        rollup(maxfloats),\n        rollup(maxints),\n        rollup(maxtimes),\n        rollup(minbyfloats),\n        rollup(minbyints),\n        rollup(minbytimes),\n        rollup(minfloats),\n        rollup(minints),\n        rollup(mintimes),\n        arrow_max_int_into_array(maxints,accessorintoarray),\n        arrow_max_int_into_values(maxints,accessorintovalues),\n        arrow_max_time_into_array(maxtimes,accessorintoarray),\n        arrow_max_time_into_values(maxtimes,accessorintovalues),\n        arrow_min_float_into_array(minfloats,accessorintoarray),\n        arrow_min_float_into_values(minfloats,accessorintovalues),\n        arrow_min_int_into_array(minints,accessorintoarray),\n        arrow_min_int_into_values(minints,accessorintovalues),\n        arrow_min_time_into_array(mintimes,accessorintoarray),\n        arrow_min_time_into_values(mintimes,accessorintovalues),\n        accessormaxfrequencyint_in(cstring),\n        accessormaxfrequencyint_out(accessormaxfrequencyint),\n        accessorminfrequencyint_in(cstring),\n        accessorminfrequencyint_out(accessorminfrequencyint),\n        accessortopn_in(cstring),\n        accessortopn_out(accessortopn),\n        accessortopncount_in(cstring),\n        accessortopncount_out(accessortopncount),\n        arrow_default_topn_bigint(spacesavingbigintaggregate,accessortopn),\n        arrow_default_topn_text(spacesavingtextaggregate,accessortopn),\n        arrow_freq_bigint_iter(spacesavingbigintaggregate,accessorintovalues),\n        arrow_freq_text_iter(spacesavingtextaggregate,accessorintovalues),\n        arrow_max_bigint_frequency(spacesavingbigintaggregate,accessormaxfrequencyint),\n        arrow_min_bigint_frequency(spacesavingbigintaggregate,accessorminfrequencyint),\n        arrow_topn_bigint(spacesavingbigintaggregate,accessortopncount),\n        arrow_topn_text(spacesavingtextaggregate,accessortopncount),\n        into_values(spacesavingaggregate,anyelement),\n        into_values(spacesavingbigintaggregate),\n        into_values(spacesavingtextaggregate),\n        max_frequency(bigint),\n        max_frequency(spacesavingaggregate,anyelement),\n        max_frequency(spacesavingbigintaggregate,bigint),\n        max_frequency(spacesavingtextaggregate,text),\n        mcv_agg(integer,bigint),\n        mcv_agg(integer,double precision,bigint),\n        mcv_agg(integer,double precision,text),\n        mcv_agg(integer,text),\n        mcv_agg_bigint_trans(internal,integer,bigint),\n        mcv_agg_text_trans(internal,integer,text),\n        mcv_agg_trans(internal,integer,anyelement),\n        mcv_agg_with_skew_bigint_trans(internal,integer,double precision,bigint),\n        mcv_agg_with_skew_text_trans(internal,integer,double precision,text),\n        mcv_agg_with_skew_trans(internal,integer,double precision,anyelement),\n        min_frequency(bigint),\n        min_frequency(spacesavingaggregate,anyelement),\n        min_frequency(spacesavingbigintaggregate,bigint),\n        min_frequency(spacesavingtextaggregate,text),\n        raw_mcv_agg(integer,anyelement),\n        raw_mcv_agg(integer,double precision,anyelement),\n        rollup(spacesavingaggregate),\n        rollup(spacesavingbigintaggregate),\n        rollup(spacesavingtextaggregate),\n        rollup_agg_bigint_trans(internal,spacesavingbigintaggregate),\n        rollup_agg_text_trans(internal,spacesavingtextaggregate),\n        rollup_agg_trans(internal,spacesavingaggregate),\n        space_saving_bigint_final(internal),\n        space_saving_combine(internal,internal),\n        space_saving_deserialize(bytea,internal),\n        space_saving_final(internal),\n        space_saving_serialize(internal),\n        space_saving_text_final(internal),\n        spacesavingaggregate_in(cstring),\n        spacesavingaggregate_out(spacesavingaggregate),\n        spacesavingbigintaggregate_in(cstring),\n        spacesavingbigintaggregate_out(spacesavingbigintaggregate),\n        spacesavingtextaggregate_in(cstring),\n        spacesavingtextaggregate_out(spacesavingtextaggregate),\n        topn(),\n        topn(bigint),\n        topn(spacesavingaggregate,anyelement),\n        topn(spacesavingaggregate,integer,anyelement),\n        topn(spacesavingbigintaggregate),\n        topn(spacesavingbigintaggregate,integer),\n        topn(spacesavingtextaggregate),\n        topn(spacesavingtextaggregate,integer),\n    }\n    \"1.15.0\" => {\n        arrow_counter_interpolated_delta(countersummary,counterinterpolateddeltaaccessor),\n        arrow_counter_interpolated_rate(countersummary,counterinterpolatedrateaccessor),\n        arrow_time_weighted_average_interpolated_average(timeweightsummary,timeweightinterpolatedaverageaccessor),\n        counterinterpolateddeltaaccessor_in(cstring),\n        counterinterpolateddeltaaccessor_out(counterinterpolateddeltaaccessor),\n        counterinterpolatedrateaccessor_in(cstring),\n        counterinterpolatedrateaccessor_out(counterinterpolatedrateaccessor),\n        interpolated_average(timestamp with time zone,interval,timeweightsummary,timeweightsummary),\n        interpolated_delta(timestamp with time zone,interval,countersummary,countersummary),\n        interpolated_rate(timestamp with time zone,interval,countersummary,countersummary),\n        timeweightinterpolatedaverageaccessor_in(cstring),\n        timeweightinterpolatedaverageaccessor_out(timeweightinterpolatedaverageaccessor),\n        accessorintegral_in(cstring),\n        accessorintegral_out(accessorintegral),\n        arrow_time_weighted_average_integral(timeweightsummary,accessorintegral),\n        arrow_time_weighted_average_interpolated_integral(timeweightsummary,timeweightinterpolatedintegralaccessor),\n        integral(text),\n        integral(timeweightsummary,text),\n        interpolated_integral(timestamp with time zone,interval,timeweightsummary,timeweightsummary,text),\n        interpolated_integral(timeweightsummary,timestamp with time zone, interval,timeweightsummary,timeweightsummary,text),\n        timeweightinterpolatedintegralaccessor_in(cstring),\n        timeweightinterpolatedintegralaccessor_out(timeweightinterpolatedintegralaccessor),\n        dead_ranges(heartbeatagg),\n        downtime(heartbeatagg),\n        heartbeat_agg(timestamp with time zone,timestamp with time zone,interval,interval),\n        heartbeat_final(internal),\n        heartbeat_rollup_trans(internal,heartbeatagg),\n        heartbeat_trans(internal,timestamp with time zone,timestamp with time zone,interval,interval),\n        heartbeatagg_in(cstring),\n        heartbeatagg_out(heartbeatagg),\n        interpolate(heartbeatagg,heartbeatagg),\n        interpolated_downtime(heartbeatagg,heartbeatagg),\n        interpolated_uptime(heartbeatagg,heartbeatagg),\n        live_at(heartbeatagg,timestamp with time zone),\n        live_ranges(heartbeatagg),\n        rollup(heartbeatagg),\n        uptime(heartbeatagg),\n        accessordeadranges_in(cstring),\n        accessordeadranges_out(accessordeadranges),\n        accessordowntime_in(cstring),\n        accessordowntime_out(accessordowntime),\n        accessorliveat_in(cstring),\n        accessorliveat_out(accessorliveat),\n        accessorliveranges_in(cstring),\n        accessorliveranges_out(accessorliveranges),\n        accessoruptime_in(cstring),\n        accessoruptime_out(accessoruptime),\n        arrow_heartbeat_agg_dead_ranges(heartbeatagg,accessordeadranges),\n        arrow_heartbeat_agg_downtime(heartbeatagg,accessordowntime),\n        arrow_heartbeat_agg_live_at(heartbeatagg,accessorliveat),\n        arrow_heartbeat_agg_live_ranges(heartbeatagg,accessorliveranges),\n        arrow_heartbeat_agg_uptime(heartbeatagg,accessoruptime),\n        dead_ranges(),\n        downtime(),\n        live_at(timestamp with time zone),\n        live_ranges(),\n        uptime(),\n        arrow_heartbeat_agg_interpolate(heartbeatagg,heartbeatinterpolateaccessor),\n        arrow_heartbeat_agg_interpolated_downtime(heartbeatagg,heartbeatinterpolateddowntimeaccessor),\n        arrow_heartbeat_agg_interpolated_uptime(heartbeatagg,heartbeatinterpolateduptimeaccessor),\n        heartbeatinterpolateaccessor_in(cstring),\n        heartbeatinterpolateaccessor_out(heartbeatinterpolateaccessor),\n        heartbeatinterpolateddowntimeaccessor_in(cstring),\n        heartbeatinterpolateddowntimeaccessor_out(heartbeatinterpolateddowntimeaccessor),\n        heartbeatinterpolateduptimeaccessor_in(cstring),\n        heartbeatinterpolateduptimeaccessor_out(heartbeatinterpolateduptimeaccessor),\n        interpolate(heartbeatagg),\n        interpolated_downtime(heartbeatagg),\n        interpolated_uptime(heartbeatagg),\n        duration_in(stateagg,bigint),\n        duration_in(stateagg,bigint,timestamp with time zone,interval),\n        duration_in(stateagg,text),\n        duration_in(stateagg,text,timestamp with time zone,interval),\n        interpolated_duration_in(stateagg,bigint,timestamp with time zone,interval,stateagg),\n        interpolated_duration_in(stateagg,text,timestamp with time zone,interval,stateagg),\n        interpolated_state_periods(stateagg,bigint,timestamp with time zone,interval,stateagg),\n        interpolated_state_periods(stateagg,text,timestamp with time zone,interval,stateagg),\n        interpolated_state_timeline(stateagg,timestamp with time zone,interval,stateagg),\n        interpolated_state_int_timeline(stateagg,timestamp with time zone,interval,stateagg),\n        into_int_values(stateagg),\n        into_values(stateagg),\n        rollup(stateagg),\n        state_agg(timestamp with time zone,bigint),\n        state_agg(timestamp with time zone,text),\n        state_agg_combine_fn_outer(internal,internal),\n        state_agg_deserialize_fn_outer(bytea,internal),\n        state_agg_finally_fn_outer(internal),\n        state_agg_int_trans(internal,timestamp with time zone,bigint),\n        state_agg_rollup_final(internal),\n        state_agg_rollup_trans(internal,stateagg),\n        state_agg_serialize_fn_outer(internal),\n        state_agg_transition_fn_outer(internal,timestamp with time zone,text),\n        state_at(stateagg,timestamp with time zone),\n        state_at_int(stateagg,timestamp with time zone),\n        state_int_timeline(stateagg),\n        state_periods(stateagg,bigint),\n        state_periods(stateagg,text),\n        state_timeline(stateagg),\n        stateagg_in(cstring),\n        stateagg_out(stateagg),\n        state_agg_rollup_combine(internal,internal),\n        state_agg_rollup_deserialize(bytea,internal),\n        state_agg_rollup_serialize(internal),\n        accessordurationin_in(cstring),\n        accessordurationin_out(accessordurationin),\n        accessordurationinint_in(cstring),\n        accessordurationinint_out(accessordurationinint),\n        accessordurationinrange_in(cstring),\n        accessordurationinrange_out(accessordurationinrange),\n        accessordurationinrangeint_in(cstring),\n        accessordurationinrangeint_out(accessordurationinrangeint),\n        accessorinterpolateddurationin_in(cstring),\n        accessorinterpolateddurationin_out(accessorinterpolateddurationin),\n        accessorinterpolateddurationinint_in(cstring),\n        accessorinterpolateddurationinint_out(accessorinterpolateddurationinint),\n        accessorinterpolatedstateinttimeline_in(cstring),\n        accessorinterpolatedstateinttimeline_out(accessorinterpolatedstateinttimeline),\n        accessorinterpolatedstateperiods_in(cstring),\n        accessorinterpolatedstateperiods_out(accessorinterpolatedstateperiods),\n        accessorinterpolatedstateperiodsint_in(cstring),\n        accessorinterpolatedstateperiodsint_out(accessorinterpolatedstateperiodsint),\n        accessorinterpolatedstatetimeline_in(cstring),\n        accessorinterpolatedstatetimeline_out(accessorinterpolatedstatetimeline),\n        accessorintointvalues_in(cstring),\n        accessorintointvalues_out(accessorintointvalues),\n        accessorintovalues_in(cstring),\n        accessorintovalues_out(accessorintovalues),\n        accessorstateat_in(cstring),\n        accessorstateat_out(accessorstateat),\n        accessorstateatint_in(cstring),\n        accessorstateatint_out(accessorstateatint),\n        accessorstateinttimeline_in(cstring),\n        accessorstateinttimeline_out(accessorstateinttimeline),\n        accessorstateperiods_in(cstring),\n        accessorstateperiods_out(accessorstateperiods),\n        accessorstateperiodsint_in(cstring),\n        accessorstateperiodsint_out(accessorstateperiodsint),\n        accessorstatetimeline_in(cstring),\n        accessorstatetimeline_out(accessorstatetimeline),\n        arrow_state_agg_duration_in_int(stateagg,accessordurationinint),\n        arrow_state_agg_duration_in_range_int(stateagg,accessordurationinrangeint),\n        arrow_state_agg_duration_in_range_string(stateagg,accessordurationinrange),\n        arrow_state_agg_duration_in_string(stateagg,accessordurationin),\n        arrow_state_agg_interpolated_duration_in_int(stateagg,accessorinterpolateddurationinint),\n        arrow_state_agg_interpolated_duration_in_string(stateagg,accessorinterpolateddurationin),\n        arrow_state_agg_interpolated_state_int_timeline(stateagg,accessorinterpolatedstateinttimeline),\n        arrow_state_agg_interpolated_state_periods_int(stateagg,accessorinterpolatedstateperiodsint),\n        arrow_state_agg_interpolated_state_periods_string(stateagg,accessorinterpolatedstateperiods),\n        arrow_state_agg_interpolated_state_timeline(stateagg,accessorinterpolatedstatetimeline),\n        arrow_state_agg_into_int_values(stateagg,accessorintointvalues),\n        arrow_state_agg_into_values(stateagg,accessorintovalues),\n        arrow_state_agg_state_at_int(stateagg,accessorstateatint),\n        arrow_state_agg_state_at_string(stateagg,accessorstateat),\n        arrow_state_agg_state_int_timeline(stateagg,accessorstateinttimeline),\n        arrow_state_agg_state_periods_int(stateagg,accessorstateperiodsint),\n        arrow_state_agg_state_periods_string(stateagg,accessorstateperiods),\n        arrow_state_agg_state_timeline(stateagg,accessorstatetimeline),\n        duration_in(bigint),\n        duration_in(bigint,timestamp with time zone,interval),\n        duration_in(text),\n        duration_in(text,timestamp with time zone,interval),\n        interpolated_duration_in(bigint,timestamp with time zone,interval,stateagg),\n        interpolated_duration_in(text,timestamp with time zone,interval,stateagg),\n        interpolated_state_int_timeline(timestamp with time zone,interval,stateagg),\n        interpolated_state_periods(bigint,timestamp with time zone,interval,stateagg),\n        interpolated_state_periods(text,timestamp with time zone,interval,stateagg),\n        interpolated_state_timeline(timestamp with time zone,interval,stateagg),\n        into_int_values(),\n        into_values(),\n        state_at(timestamp with time zone),\n        state_at_int(timestamp with time zone),\n        state_int_timeline(),\n        state_periods(bigint),\n        state_periods(text),\n        state_timeline(),\n    }\n    \"1.14.0\" => {\n        interpolated_average(timeweightsummary,timestamp with time zone,interval,timeweightsummary,timeweightsummary),\n        interpolated_delta(countersummary,timestamp with time zone,interval,countersummary,countersummary),\n        interpolated_rate(countersummary,timestamp with time zone,interval,countersummary,countersummary),\n        accessorclose_in(cstring),\n        accessorclose_out(accessorclose),\n        accessorclosetime_in(cstring),\n        accessorclosetime_out(accessorclosetime),\n        accessorhigh_in(cstring),\n        accessorhigh_out(accessorhigh),\n        accessorhightime_in(cstring),\n        accessorhightime_out(accessorhightime),\n        accessorlow_in(cstring),\n        accessorlow_out(accessorlow),\n        accessorlowtime_in(cstring),\n        accessorlowtime_out(accessorlowtime),\n        accessoropen_in(cstring),\n        accessoropen_out(accessoropen),\n        accessoropentime_in(cstring),\n        accessoropentime_out(accessoropentime),\n        arrow_close(candlestick,accessorclose),\n        arrow_close_time(candlestick,accessorclosetime),\n        arrow_high(candlestick,accessorhigh),\n        arrow_high_time(candlestick,accessorhightime),\n        arrow_low(candlestick,accessorlow),\n        arrow_low_time(candlestick,accessorlowtime),\n        arrow_open(candlestick,accessoropen),\n        arrow_open_time(candlestick,accessoropentime),\n        candlestick(timestamp with time zone, double precision,double precision, double precision, double precision, double precision),\n        candlestick_agg(timestamp with time zone,double precision,double precision),\n        candlestick_combine(internal,internal),\n        candlestick_deserialize(bytea,internal),\n        candlestick_final(internal),\n        candlestick_in(cstring),\n        candlestick_out(candlestick),\n        candlestick_rollup_trans(internal,candlestick),\n        candlestick_serialize(internal),\n        open(),\n        open_time(),\n        close(candlestick),\n        close(),\n        close_time(candlestick),\n        close_time(),\n        high(candlestick),\n        high(),\n        high_time(candlestick),\n        high_time(),\n        low(candlestick),\n        low(),\n        low_time(candlestick),\n        low_time(),\n        open(candlestick),\n        open_time(candlestick),\n        rollup(candlestick),\n        tick_data_no_vol_transition(internal,timestamp with time zone,double precision),\n        tick_data_transition(internal,timestamp with time zone,double precision,double precision),\n        volume(candlestick),\n        vwap(candlestick),\n    }\n    \"1.12.0\" => {\n        stats1d_tf_inv_trans(internal,double precision),\n        stats1d_tf_final(internal),\n        stats1d_tf_trans(internal,double precision),\n        stats2d_tf_final(internal),\n        stats2d_tf_trans(internal,double precision,double precision),\n        stats2d_tf_inv_trans(internal,double precision,double precision),\n    }\n    \"1.11.0\" => {\n        accessorfirsttime_in(cstring),\n        accessorfirsttime_out(accessorfirsttime),\n        accessorfirstval_in(cstring),\n        accessorfirstval_out(accessorfirstval),\n        accessorlasttime_in(cstring),\n        accessorlasttime_out(accessorlasttime),\n        accessorlastval_in(cstring),\n        accessorlastval_out(accessorlastval),\n        arrow_counter_agg_first_time(countersummary,accessorfirsttime),\n        arrow_counter_agg_first_val(countersummary,accessorfirstval),\n        arrow_counter_agg_last_time(countersummary,accessorlasttime),\n        arrow_counter_agg_last_val(countersummary,accessorlastval),\n        arrow_time_weight_first_time(timeweightsummary,accessorfirsttime),\n        arrow_time_weight_first_val(timeweightsummary,accessorfirstval),\n        arrow_time_weight_last_time(timeweightsummary,accessorlasttime),\n        arrow_time_weight_last_val(timeweightsummary,accessorlastval),\n        first_time(),\n        first_time(countersummary),\n        first_time(timeweightsummary),\n        first_val(),\n        first_val(countersummary),\n        first_val(timeweightsummary),\n        last_time(),\n        last_time(countersummary),\n        last_time(timeweightsummary),\n        last_val(),\n        last_val(countersummary),\n        last_val(timeweightsummary),\n        asap_final(internal),\n        asap_smooth(timestamp with time zone,double precision,integer),\n        asap_smooth(timevector_tstz_f64,integer),\n        asap_trans(internal,timestamp with time zone,double precision,integer),\n    }\n    \"1.9.0\" => {\n        accessorapproxpercentile_in(cstring),\n        accessorapproxpercentile_out(accessorapproxpercentile),\n        accessorapproxpercentilerank_in(cstring),\n        accessorapproxpercentilerank_out(accessorapproxpercentilerank),\n        accessoraverage_in(cstring),\n        accessoraverage_out(accessoraverage),\n        accessoraveragex_in(cstring),\n        accessoraveragex_out(accessoraveragex),\n        accessoraveragey_in(cstring),\n        accessoraveragey_out(accessoraveragey),\n        accessorcorr_in(cstring),\n        accessorcorr_out(accessorcorr),\n        accessorcounterzerotime_in(cstring),\n        accessorcounterzerotime_out(accessorcounterzerotime),\n        accessorcovar_in(cstring),\n        accessorcovar_out(accessorcovar),\n        accessordelta_in(cstring),\n        accessordelta_out(accessordelta),\n        accessordeterminationcoeff_in(cstring),\n        accessordeterminationcoeff_out(accessordeterminationcoeff),\n        accessordistinctcount_in(cstring),\n        accessordistinctcount_out(accessordistinctcount),\n        accessorerror_in(cstring),\n        accessorerror_out(accessorerror),\n        accessorextrapolateddelta_in(cstring),\n        accessorextrapolateddelta_out(accessorextrapolateddelta),\n        accessorextrapolatedrate_in(cstring),\n        accessorextrapolatedrate_out(accessorextrapolatedrate),\n        accessorideltaleft_in(cstring),\n        accessorideltaleft_out(accessorideltaleft),\n        accessorideltaright_in(cstring),\n        accessorideltaright_out(accessorideltaright),\n        accessorintercept_in(cstring),\n        accessorintercept_out(accessorintercept),\n        accessorirateleft_in(cstring),\n        accessorirateleft_out(accessorirateleft),\n        accessorirateright_in(cstring),\n        accessorirateright_out(accessorirateright),\n        accessorkurtosis_in(cstring),\n        accessorkurtosis_out(accessorkurtosis),\n        accessorkurtosisx_in(cstring),\n        accessorkurtosisx_out(accessorkurtosisx),\n        accessorkurtosisy_in(cstring),\n        accessorkurtosisy_out(accessorkurtosisy),\n        accessormaxval_in(cstring),\n        accessormaxval_out(accessormaxval),\n        accessormean_in(cstring),\n        accessormean_out(accessormean),\n        accessorminval_in(cstring),\n        accessorminval_out(accessorminval),\n        accessornumchanges_in(cstring),\n        accessornumchanges_out(accessornumchanges),\n        accessornumelements_in(cstring),\n        accessornumelements_out(accessornumelements),\n        accessornumresets_in(cstring),\n        accessornumresets_out(accessornumresets),\n        accessornumvals_in(cstring),\n        accessornumvals_out(accessornumvals),\n        accessorrate_in(cstring),\n        accessorrate_out(accessorrate),\n        accessorskewness_in(cstring),\n        accessorskewness_out(accessorskewness),\n        accessorskewnessx_in(cstring),\n        accessorskewnessx_out(accessorskewnessx),\n        accessorskewnessy_in(cstring),\n        accessorskewnessy_out(accessorskewnessy),\n        accessorslope_in(cstring),\n        accessorslope_out(accessorslope),\n        accessorstddev_in(cstring),\n        accessorstddev_out(accessorstddev),\n        accessorstddevx_in(cstring),\n        accessorstddevx_out(accessorstddevx),\n        accessorstddevy_in(cstring),\n        accessorstddevy_out(accessorstddevy),\n        accessorstderror_in(cstring),\n        accessorstderror_out(accessorstderror),\n        accessorsum_in(cstring),\n        accessorsum_out(accessorsum),\n        accessorsumx_in(cstring),\n        accessorsumx_out(accessorsumx),\n        accessorsumy_in(cstring),\n        accessorsumy_out(accessorsumy),\n        accessortimedelta_in(cstring),\n        accessortimedelta_out(accessortimedelta),\n        accessorunnest_in(cstring),\n        accessorunnest_out(accessorunnest),\n        accessorvariance_in(cstring),\n        accessorvariance_out(accessorvariance),\n        accessorvariancex_in(cstring),\n        accessorvariancex_out(accessorvariancex),\n        accessorvariancey_in(cstring),\n        accessorvariancey_out(accessorvariancey),\n        accessorwithbounds_in(cstring),\n        accessorwithbounds_out(accessorwithbounds),\n        accessorxintercept_in(cstring),\n        accessorxintercept_out(accessorxintercept),\n        approx_percentile(double precision),\n        approx_percentile_rank(double precision),\n        arrow_counter_agg_corr(countersummary,accessorcorr),\n        arrow_counter_agg_delta(countersummary,accessordelta),\n        arrow_counter_agg_extrapolated_delta(countersummary,accessorextrapolateddelta),\n        arrow_counter_agg_extrapolated_rate(countersummary,accessorextrapolatedrate),\n        arrow_counter_agg_idelta_left(countersummary,accessorideltaleft),\n        arrow_counter_agg_idelta_right(countersummary,accessorideltaright),\n        arrow_counter_agg_intercept(countersummary,accessorintercept),\n        arrow_counter_agg_irate_left(countersummary,accessorirateleft),\n        arrow_counter_agg_irate_right(countersummary,accessorirateright),\n        arrow_counter_agg_num_changes(countersummary,accessornumchanges),\n        arrow_counter_agg_num_elements(countersummary,accessornumelements),\n        arrow_counter_agg_num_resets(countersummary,accessornumresets),\n        arrow_counter_agg_rate(countersummary,accessorrate),\n        arrow_counter_agg_slope(countersummary,accessorslope),\n        arrow_counter_agg_time_delta(countersummary,accessortimedelta),\n        arrow_counter_agg_with_bounds(countersummary,accessorwithbounds),\n        arrow_counter_agg_zero_time(countersummary,accessorcounterzerotime),\n        arrow_hyperloglog_count(hyperloglog,accessordistinctcount),\n        arrow_hyperloglog_error(hyperloglog,accessorstderror),\n        arrow_stats1d_average(statssummary1d,accessoraverage),\n        arrow_stats1d_kurtosis(statssummary1d,accessorkurtosis),\n        arrow_stats1d_num_vals(statssummary1d,accessornumvals),\n        arrow_stats1d_skewness(statssummary1d,accessorskewness),\n        arrow_stats1d_stddev(statssummary1d,accessorstddev),\n        arrow_stats1d_sum(statssummary1d,accessorsum),\n        arrow_stats1d_variance(statssummary1d,accessorvariance),\n        arrow_stats2d_average_x(statssummary2d,accessoraveragex),\n        arrow_stats2d_average_y(statssummary2d,accessoraveragey),\n        arrow_stats2d_corr(statssummary2d,accessorcorr),\n        arrow_stats2d_covar(statssummary2d,accessorcovar),\n        arrow_stats2d_determination_coeff(statssummary2d,accessordeterminationcoeff),\n        arrow_stats2d_intercept(statssummary2d,accessorintercept),\n        arrow_stats2d_kurtosis_x(statssummary2d,accessorkurtosisx),\n        arrow_stats2d_kurtosis_y(statssummary2d,accessorkurtosisy),\n        arrow_stats2d_num_vals(statssummary2d,accessornumvals),\n        arrow_stats2d_skewness_x(statssummary2d,accessorskewnessx),\n        arrow_stats2d_skewness_y(statssummary2d,accessorskewnessy),\n        arrow_stats2d_slope(statssummary2d,accessorslope),\n        arrow_stats2d_stdddev_x(statssummary2d,accessorstddevx),\n        arrow_stats2d_stdddev_y(statssummary2d,accessorstddevy),\n        arrow_stats2d_sum_x(statssummary2d,accessorsumx),\n        arrow_stats2d_sum_y(statssummary2d,accessorsumy),\n        arrow_stats2d_variance_x(statssummary2d,accessorvariancex),\n        arrow_stats2d_variance_y(statssummary2d,accessorvariancey),\n        arrow_stats2d_x_intercept(statssummary2d,accessorxintercept),\n        arrow_tdigest_approx_percentile(tdigest,accessorapproxpercentile),\n        arrow_tdigest_approx_rank(tdigest,accessorapproxpercentilerank),\n        arrow_tdigest_max(tdigest,accessormaxval),\n        arrow_tdigest_mean(tdigest,accessormean),\n        arrow_tdigest_min(tdigest,accessorminval),\n        arrow_tdigest_num_vals(tdigest,accessornumvals),\n        arrow_time_weighted_average_average(timeweightsummary,accessoraverage),\n        arrow_timevector_unnest(timevector_tstz_f64,accessorunnest),\n        arrow_uddsketch_approx_percentile(uddsketch,accessorapproxpercentile),\n        arrow_uddsketch_approx_rank(uddsketch,accessorapproxpercentilerank),\n        arrow_uddsketch_error(uddsketch,accessorerror),\n        arrow_uddsketch_mean(uddsketch,accessormean),\n        arrow_uddsketch_num_vals(uddsketch,accessornumvals),\n        average(),\n        average_x(),\n        average_y(),\n        corr(),\n        counter_zero_time(),\n        covariance(text),\n        delta(),\n        determination_coeff(),\n        distinct_count(),\n        error(),\n        extrapolated_delta(text),\n        extrapolated_rate(text),\n        idelta_left(),\n        idelta_right(),\n        intercept(),\n        irate_left(),\n        irate_right(),\n        kurtosis(text),\n        kurtosis_x(text),\n        kurtosis_y(text),\n        lttb(timestamp with time zone,double precision,integer),\n        lttb(timevector_tstz_f64,integer),\n        lttb_final(internal),\n        lttb_trans(internal,timestamp with time zone,double precision,integer),\n        max_val(),\n        mean(),\n        min_val(),\n        num_changes(),\n        num_elements(),\n        num_resets(),\n        num_vals(),\n        rate(),\n        rollup(timevector_tstz_f64),\n        skewness(text),\n        skewness_x(text),\n        skewness_y(text),\n        slope(),\n        stddev(text),\n        stddev_x(text),\n        stddev_y(text),\n        stderror(),\n        sum(),\n        sum_x(),\n        sum_y(),\n        time_delta(),\n        timevector(timestamp with time zone,double precision),\n        timevector_combine(internal,internal),\n        timevector_deserialize(bytea,internal),\n        timevector_final(internal),\n        timevector_serialize(internal),\n        timevector_tstz_f64_compound_trans(internal,timevector_tstz_f64),\n        timevector_tstz_f64_in(cstring),\n        timevector_tstz_f64_out(timevector_tstz_f64),\n        timevector_tstz_f64_trans(internal,timestamp with time zone,double precision),\n        unnest(),\n        unnest(timevector_tstz_f64),\n        variance(text),\n        variance_x(text),\n        variance_y(text),\n        with_bounds(tstzrange),\n        x_intercept(),\n        lttb(timestamp with time zone,double precision,integer),\n        lttb(timevector_tstz_f64,integer),\n        lttb_final(internal),\n        lttb_trans(internal,timestamp with time zone,double precision,integer),\n    }\n    \"1.8.0\" => {\n    }\n    \"1.7.0\" => {\n    }\n    \"1.6.0\" => {\n    }\n    \"1.5\" => {\n    }\n    \"prehistory\" => {\n        approx_percentile(double precision,uddsketch),\n        approx_percentile_rank(double precision,uddsketch),\n        error(uddsketch),\n        mean(uddsketch),\n        num_vals(uddsketch),\n        percentile_agg(double precision),\n        percentile_agg_trans(internal,double precision),\n        uddsketch(integer,double precision,double precision),\n        rollup(uddsketch),\n        uddsketch_combine(internal,internal),\n        uddsketch_compound_trans(internal,uddsketch),\n        uddsketch_deserialize(bytea,internal),\n        uddsketch_final(internal),\n        uddsketch_in(cstring),\n        uddsketch_out(uddsketch),\n        uddsketch_serialize(internal),\n        uddsketch_trans(internal,integer,double precision,double precision),\n        approx_percentile(double precision,tdigest),\n        approx_percentile_rank(double precision,tdigest),\n        max_val(tdigest),\n        min_val(tdigest),\n        mean(tdigest),\n        num_vals(tdigest),\n        tdigest(integer,double precision),\n        rollup(tdigest),\n        tdigest_combine(internal,internal),\n        tdigest_compound_combine(internal,internal),\n        tdigest_compound_deserialize(bytea,internal),\n        tdigest_compound_final(internal),\n        tdigest_compound_serialize(internal),\n        tdigest_compound_trans(internal,tdigest),\n        tdigest_deserialize(bytea,internal),\n        tdigest_final(internal),\n        tdigest_in(cstring),\n        tdigest_out(tdigest),\n        tdigest_serialize(internal),\n        tdigest_trans(internal,integer,double precision),\n        average(timeweightsummary),\n        time_weight(text,timestamp with time zone,double precision),\n        rollup(timeweightsummary),\n        time_weight_combine(internal,internal),\n        time_weight_final(internal),\n        time_weight_summary_trans(internal,timeweightsummary),\n        time_weight_trans(internal,text,timestamp with time zone,double precision),\n        time_weight_trans_deserialize(bytea,internal),\n        time_weight_trans_serialize(internal),\n        timeweightsummary_in(cstring),\n        timeweightsummary_out(timeweightsummary),\n        corr(countersummary),\n        counter_agg(timestamp with time zone,double precision),\n        counter_agg(timestamp with time zone,double precision,tstzrange),\n        counter_agg_combine(internal,internal),\n        counter_agg_final(internal),\n        counter_agg_summary_trans(internal,countersummary),\n        counter_agg_trans(internal,timestamp with time zone,double precision,tstzrange),\n        counter_agg_trans_no_bounds(internal,timestamp with time zone,double precision),\n        counter_summary_trans_deserialize(bytea,internal),\n        counter_summary_trans_serialize(internal),\n        counter_zero_time(countersummary),\n        countersummary_in(cstring),\n        countersummary_out(countersummary),\n        delta(countersummary),\n        extrapolated_delta(countersummary,text),\n        extrapolated_rate(countersummary,text),\n        idelta_left(countersummary),\n        idelta_right(countersummary),\n        intercept(countersummary),\n        irate_left(countersummary),\n        irate_right(countersummary),\n        num_changes(countersummary),\n        num_elements(countersummary),\n        num_resets(countersummary),\n        rate(countersummary),\n        rollup(countersummary),\n        slope(countersummary),\n        time_delta(countersummary),\n        with_bounds(countersummary,tstzrange),\n        hyperloglog(integer,anyelement),\n        hyperloglog_combine(internal,internal),\n        hyperloglog_deserialize(bytea,internal),\n        hyperloglog_final(internal),\n        hyperloglog_in(cstring),\n        hyperloglog_out(hyperloglog),\n        hyperloglog_serialize(internal),\n        hyperloglog_trans(internal,integer,anyelement),\n        hyperloglog_union(internal,hyperloglog),\n        rollup(hyperloglog),\n        stderror(hyperloglog),\n        average(statssummary1d),\n        average_x(statssummary2d),\n        average_y(statssummary2d),\n        corr(statssummary2d),\n        covariance(statssummary2d,text),\n        determination_coeff(statssummary2d),\n        intercept(statssummary2d),\n        kurtosis(statssummary1d,text),\n        kurtosis_x(statssummary2d,text),\n        kurtosis_y(statssummary2d,text),\n        num_vals(statssummary1d),\n        num_vals(statssummary2d),\n        rolling(statssummary1d),\n        rolling(statssummary2d),\n        rollup(statssummary1d),\n        rollup(statssummary2d),\n        skewness(statssummary1d,text),\n        skewness_x(statssummary2d,text),\n        skewness_y(statssummary2d,text),\n        slope(statssummary2d),\n        stats1d_combine(internal,internal),\n        stats1d_final(internal),\n        stats1d_inv_trans(internal,double precision),\n        stats1d_summary_inv_trans(internal,statssummary1d),\n        stats1d_summary_trans(internal,statssummary1d),\n        stats1d_trans(internal,double precision),\n        stats1d_trans_deserialize(bytea,internal),\n        stats1d_trans_serialize(internal),\n        stats2d_combine(internal,internal),\n        stats2d_final(internal),\n        stats2d_inv_trans(internal,double precision,double precision),\n        stats2d_summary_inv_trans(internal,statssummary2d),\n        stats2d_summary_trans(internal,statssummary2d),\n        stats2d_trans(internal,double precision,double precision),\n        stats2d_trans_deserialize(bytea,internal),\n        stats2d_trans_serialize(internal),\n        stats_agg(double precision),\n        stats_agg(double precision,double precision),\n        stats_agg_no_inv(double precision),\n        stats_agg_no_inv(double precision,double precision),\n        statssummary1d_in(cstring),\n        statssummary1d_out(statssummary1d),\n        statssummary2d_in(cstring),\n        statssummary2d_out(statssummary2d),\n        stddev(statssummary1d,text),\n        stddev_x(statssummary2d,text),\n        stddev_y(statssummary2d,text),\n        sum(statssummary1d),\n        sum_x(statssummary2d),\n        sum_y(statssummary2d),\n        variance(statssummary1d,text),\n        variance_x(statssummary2d,text),\n        variance_y(statssummary2d,text),\n        x_intercept(statssummary2d),\n        distinct_count(hyperloglog),\n    }\n}\n\ncrate::types_stabilized_at! {\n    STABLE_TYPES\n    \"1.16.0\" => {\n        accessornumgaps,\n        accessornumliveranges,\n        heartbeattrimtoaccessor,\n        accessorpercentilearray,\n        maxbyfloats,\n        maxbyints,\n        maxbytimes,\n        maxfloats,\n        maxints,\n        maxtimes,\n        minbyfloats,\n        minbyints,\n        minbytimes,\n        minfloats,\n        minints,\n        mintimes,\n        accessorintoarray,\n        accessormaxfrequencyint,\n        accessorminfrequencyint,\n        accessortopn,\n        accessortopncount,\n        spacesavingaggregate,\n        spacesavingbigintaggregate,\n        spacesavingtextaggregate,\n    }\n    \"1.15.0\" => {\n        counterinterpolateddeltaaccessor,\n        counterinterpolatedrateaccessor,\n        timeweightinterpolatedaverageaccessor,\n        timeweightinterpolatedintegralaccessor,\n        accessorintegral,\n        heartbeatagg,\n        accessordeadranges,\n        accessordowntime,\n        accessorliveat,\n        accessorliveranges,\n        accessoruptime,\n        heartbeatinterpolateaccessor,\n        heartbeatinterpolateddowntimeaccessor,\n        heartbeatinterpolateduptimeaccessor,\n        stateagg,\n        accessordurationin,\n        accessordurationinint,\n        accessordurationinrange,\n        accessordurationinrangeint,\n        accessorinterpolateddurationin,\n        accessorinterpolateddurationinint,\n        accessorinterpolatedstateinttimeline,\n        accessorinterpolatedstateperiods,\n        accessorinterpolatedstateperiodsint,\n        accessorinterpolatedstatetimeline,\n        accessorintointvalues,\n        accessorintovalues,\n        accessorstateat,\n        accessorstateatint,\n        accessorstateinttimeline,\n        accessorstateperiods,\n        accessorstateperiodsint,\n        accessorstatetimeline,\n    }\n    \"1.14.0\" => {\n        candlestick,\n        accessorclose,\n        accessorclosetime,\n        accessorhigh,\n        accessorhightime,\n        accessorlow,\n        accessorlowtime,\n        accessoropen,\n        accessoropentime,\n    }\n    \"1.11.0\" => {\n        accessorfirsttime,\n        accessorfirstval,\n        accessorlasttime,\n        accessorlastval,\n    }\n    \"1.9.0\" => {\n        accessorapproxpercentile,\n        accessorapproxpercentilerank,\n        accessoraverage,\n        accessoraveragex,\n        accessoraveragey,\n        accessorcorr,\n        accessorcounterzerotime,\n        accessorcovar,\n        accessordelta,\n        accessordeterminationcoeff,\n        accessordistinctcount,\n        accessorerror,\n        accessorextrapolateddelta,\n        accessorextrapolatedrate,\n        accessorideltaleft,\n        accessorideltaright,\n        accessorintercept,\n        accessorirateleft,\n        accessorirateright,\n        accessorkurtosis,\n        accessorkurtosisx,\n        accessorkurtosisy,\n        accessormaxval,\n        accessormean,\n        accessorminval,\n        accessornumchanges,\n        accessornumelements,\n        accessornumresets,\n        accessornumvals,\n        accessorrate,\n        accessorskewness,\n        accessorskewnessx,\n        accessorskewnessy,\n        accessorslope,\n        accessorstddev,\n        accessorstddevx,\n        accessorstddevy,\n        accessorstderror,\n        accessorsum,\n        accessorsumx,\n        accessorsumy,\n        accessortimedelta,\n        accessorunnest,\n        accessorvariance,\n        accessorvariancex,\n        accessorvariancey,\n        accessorwithbounds,\n        accessorxintercept,\n        timevector_tstz_f64,\n    }\n    \"1.8.0\" => {\n    }\n    \"1.7.0\" => {\n    }\n    \"1.6.0\" => {\n    }\n    \"1.5\" => {\n    }\n    \"prehistory\" => {\n        uddsketch,\n        tdigest,\n        timeweightsummary,\n        countersummary,\n        hyperloglog,\n        statssummary1d,\n        statssummary2d,\n    }\n}\n\ncrate::operators_stabilized_at! {\n    STABLE_OPERATORS\n    \"1.16.0\" => {\n        \"->\"(heartbeatagg,accessornumgaps),\n        \"->\"(heartbeatagg,accessornumliveranges),\n        \"->\"(heartbeatagg,heartbeattrimtoaccessor),\n        \"->\"(uddsketch,accessorpercentilearray),\n        \"->\"(maxfloats,accessorintoarray),\n        \"->\"(maxfloats,accessorintovalues),\n        \"->\"(maxints,accessorintoarray),\n        \"->\"(maxints,accessorintovalues),\n        \"->\"(maxtimes,accessorintoarray),\n        \"->\"(maxtimes,accessorintovalues),\n        \"->\"(minfloats,accessorintoarray),\n        \"->\"(minfloats,accessorintovalues),\n        \"->\"(minints,accessorintoarray),\n        \"->\"(minints,accessorintovalues),\n        \"->\"(mintimes,accessorintoarray),\n        \"->\"(mintimes,accessorintovalues),\n        \"->\"(spacesavingbigintaggregate,accessorintovalues),\n        \"->\"(spacesavingbigintaggregate,accessormaxfrequencyint),\n        \"->\"(spacesavingbigintaggregate,accessorminfrequencyint),\n        \"->\"(spacesavingbigintaggregate,accessortopn),\n        \"->\"(spacesavingbigintaggregate,accessortopncount),\n        \"->\"(spacesavingtextaggregate,accessorintovalues),\n        \"->\"(spacesavingtextaggregate,accessortopn),\n        \"->\"(spacesavingtextaggregate,accessortopncount),\n    }\n    \"1.15.0\" => {\n        \"->\"(countersummary,counterinterpolateddeltaaccessor),\n        \"->\"(countersummary,counterinterpolatedrateaccessor),\n        \"->\"(timeweightsummary,timeweightinterpolatedaverageaccessor),\n        \"->\"(timeweightsummary,timeweightinterpolatedintegralaccessor),\n        \"->\"(timeweightsummary,accessorintegral),\n        \"->\"(heartbeatagg,accessordeadranges),\n        \"->\"(heartbeatagg,accessordowntime),\n        \"->\"(heartbeatagg,accessorliveat),\n        \"->\"(heartbeatagg,accessorliveranges),\n        \"->\"(heartbeatagg,accessoruptime),\n        \"->\"(heartbeatagg,heartbeatinterpolateaccessor),\n        \"->\"(heartbeatagg,heartbeatinterpolateddowntimeaccessor),\n        \"->\"(heartbeatagg,heartbeatinterpolateduptimeaccessor),\n        \"->\"(stateagg,accessordurationin),\n        \"->\"(stateagg,accessordurationinint),\n        \"->\"(stateagg,accessordurationinrange),\n        \"->\"(stateagg,accessordurationinrangeint),\n        \"->\"(stateagg,accessorinterpolateddurationin),\n        \"->\"(stateagg,accessorinterpolateddurationinint),\n        \"->\"(stateagg,accessorinterpolatedstateinttimeline),\n        \"->\"(stateagg,accessorinterpolatedstateperiods),\n        \"->\"(stateagg,accessorinterpolatedstateperiodsint),\n        \"->\"(stateagg,accessorinterpolatedstatetimeline),\n        \"->\"(stateagg,accessorintointvalues),\n        \"->\"(stateagg,accessorintovalues),\n        \"->\"(stateagg,accessorstateat),\n        \"->\"(stateagg,accessorstateatint),\n        \"->\"(stateagg,accessorstateinttimeline),\n        \"->\"(stateagg,accessorstateperiods),\n        \"->\"(stateagg,accessorstateperiodsint),\n        \"->\"(stateagg,accessorstatetimeline),\n    }\n    \"1.14.0\" => {\n        \"->\"(candlestick,accessorclose),\n        \"->\"(candlestick,accessorclosetime),\n        \"->\"(candlestick,accessorhigh),\n        \"->\"(candlestick,accessorhightime),\n        \"->\"(candlestick,accessorlow),\n        \"->\"(candlestick,accessorlowtime),\n        \"->\"(candlestick,accessoropen),\n        \"->\"(candlestick,accessoropentime),\n    }\n    \"1.11.0\" => {\n        \"->\"(countersummary,accessorfirsttime),\n        \"->\"(countersummary,accessorfirstval),\n        \"->\"(countersummary,accessorlasttime),\n        \"->\"(countersummary,accessorlastval),\n        \"->\"(timeweightsummary,accessorfirsttime),\n        \"->\"(timeweightsummary,accessorfirstval),\n        \"->\"(timeweightsummary,accessorlasttime),\n        \"->\"(timeweightsummary,accessorlastval),\n    }\n    \"1.9.0\" => {\n        \"->\"(countersummary,accessorcorr),\n        \"->\"(countersummary,accessorcounterzerotime),\n        \"->\"(countersummary,accessordelta),\n        \"->\"(countersummary,accessorextrapolateddelta),\n        \"->\"(countersummary,accessorextrapolatedrate),\n        \"->\"(countersummary,accessorideltaleft),\n        \"->\"(countersummary,accessorideltaright),\n        \"->\"(countersummary,accessorintercept),\n        \"->\"(countersummary,accessorirateleft),\n        \"->\"(countersummary,accessorirateright),\n        \"->\"(countersummary,accessornumchanges),\n        \"->\"(countersummary,accessornumelements),\n        \"->\"(countersummary,accessornumresets),\n        \"->\"(countersummary,accessorrate),\n        \"->\"(countersummary,accessorslope),\n        \"->\"(countersummary,accessortimedelta),\n        \"->\"(countersummary,accessorwithbounds),\n        \"->\"(hyperloglog,accessordistinctcount),\n        \"->\"(hyperloglog,accessorstderror),\n        \"->\"(statssummary1d,accessoraverage),\n        \"->\"(statssummary1d,accessorkurtosis),\n        \"->\"(statssummary1d,accessornumvals),\n        \"->\"(statssummary1d,accessorskewness),\n        \"->\"(statssummary1d,accessorstddev),\n        \"->\"(statssummary1d,accessorsum),\n        \"->\"(statssummary1d,accessorvariance),\n        \"->\"(statssummary2d,accessoraveragex),\n        \"->\"(statssummary2d,accessoraveragey),\n        \"->\"(statssummary2d,accessorcorr),\n        \"->\"(statssummary2d,accessorcovar),\n        \"->\"(statssummary2d,accessordeterminationcoeff),\n        \"->\"(statssummary2d,accessorintercept),\n        \"->\"(statssummary2d,accessorkurtosisx),\n        \"->\"(statssummary2d,accessorkurtosisy),\n        \"->\"(statssummary2d,accessornumvals),\n        \"->\"(statssummary2d,accessorskewnessx),\n        \"->\"(statssummary2d,accessorskewnessy),\n        \"->\"(statssummary2d,accessorslope),\n        \"->\"(statssummary2d,accessorstddevx),\n        \"->\"(statssummary2d,accessorstddevy),\n        \"->\"(statssummary2d,accessorsumx),\n        \"->\"(statssummary2d,accessorsumy),\n        \"->\"(statssummary2d,accessorvariancex),\n        \"->\"(statssummary2d,accessorvariancey),\n        \"->\"(statssummary2d,accessorxintercept),\n        \"->\"(tdigest,accessorapproxpercentile),\n        \"->\"(tdigest,accessorapproxpercentilerank),\n        \"->\"(tdigest,accessormaxval),\n        \"->\"(tdigest,accessormean),\n        \"->\"(tdigest,accessorminval),\n        \"->\"(tdigest,accessornumvals),\n        \"->\"(timevector_tstz_f64,accessorunnest),\n        \"->\"(timeweightsummary,accessoraverage),\n        \"->\"(uddsketch,accessorapproxpercentile),\n        \"->\"(uddsketch,accessorapproxpercentilerank),\n        \"->\"(uddsketch,accessorerror),\n        \"->\"(uddsketch,accessormean),\n        \"->\"(uddsketch,accessornumvals),\n    }\n    \"1.8.0\" => {\n    }\n    \"1.7.0\" => {\n    }\n    \"1.6.0\" => {\n    }\n    \"1.5\" => {\n    }\n    \"prehistory\" => {\n    }\n}\n"
  },
  {
    "path": "extension/src/stabilization_tests.rs",
    "content": "#[cfg(any(test, feature = \"pg_test\"))]\nuse pgrx::*;\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use std::collections::HashSet;\n\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    // Test that any new features are added to the the experimental schema\n    #[pg_test]\n    fn test_schema_qualification() {\n        Spi::connect_mut(|client| {\n            let stable_functions: HashSet<String> = stable_functions();\n            let stable_types: HashSet<String> = stable_types();\n            let stable_operators: HashSet<String> = stable_operators();\n            let unexpected_features: Vec<_> = client\n                .update(\n                    \"SELECT pg_catalog.pg_describe_object(classid, objid, 0) \\\n                    FROM pg_catalog.pg_extension e, pg_catalog.pg_depend d \\\n                    WHERE e.extname='timescaledb_toolkit' \\\n                    AND refclassid = 'pg_catalog.pg_extension'::pg_catalog.regclass \\\n                    AND d.refobjid = e.oid \\\n                    AND deptype = 'e'\n                    ORDER BY 1\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .filter_map(|row| {\n                    let val: String = row\n                        .get_datum_by_ordinal(1)\n                        .unwrap()\n                        .value()\n                        .unwrap()\n                        .unwrap();\n\n                    if let Some(schema) = val.strip_prefix(\"schema \") {\n                        // the only schemas we should define are\n                        // `toolkit_experimental` our experimental schema, and\n                        // `tests` which contains our pgrx-style regression tests\n                        // (including the function currently running)\n                        match schema {\n                            \"toolkit_experimental\" => return None,\n                            \"tests\" => return None,\n                            _ => return Some(val),\n                        }\n                    }\n\n                    if let Some(ty) = val.strip_prefix(\"type \") {\n                        // types in the experimental schema are experimental\n                        if ty.starts_with(\"toolkit_experimental.\") {\n                            return None;\n                        }\n\n                        // PG17 started automatically creating an array type for types, so we need\n                        // to take those into account.\n                        let ty_no_array = ty.replace(\"[]\", \"\");\n\n                        if stable_types.contains(ty) || stable_types.contains(&ty_no_array) {\n                            return None;\n                        }\n\n                        return Some(val);\n                    }\n\n                    if let Some(function) = val.strip_prefix(\"function \") {\n                        // functions in the experimental schema are experimental\n                        if function.starts_with(\"toolkit_experimental.\") {\n                            return None;\n                        }\n\n                        // functions in test schema only exist for tests and\n                        // won't be in release versions of the extension\n                        if function.starts_with(\"tests.\") {\n                            return None;\n                        }\n\n                        // arrow functions outside the experimental schema are\n                        // considered experimental as long as one of their argument\n                        // types are experimental (`#[pg_operator]` doesn't allow\n                        // us to declare these in a schema and the operator using\n                        // them not in the schema). We use name-based resolution\n                        // to tell if a function exists to implement an arrow\n                        // operator because we didn't have a better method\n                        let is_arrow =\n                            function.starts_with(\"arrow_\") || function.starts_with(\"finalize_with\");\n                        if is_arrow && function.contains(\"toolkit_experimental.\") {\n                            return None;\n                        }\n\n                        if stable_functions.contains(function) {\n                            return None;\n                        }\n\n                        // Hack to fix the function macro's inability to handle [] in the type double precision[].\n                        if function == \"approx_percentile_array(double precision[],uddsketch)\"\n                            || function == \"approx_percentiles(double precision[])\"\n                        {\n                            return None;\n                        }\n\n                        return Some(val);\n                    }\n\n                    if let Some(operator) = val.strip_prefix(\"operator \") {\n                        // we generally don't put operators in the experimental\n                        // schema if we can avoid it because we consider the\n                        // `OPERATOR(schema.<op>)` syntax to be to much a\n                        // usability hazard. Instead we rely on one of the input\n                        // types being experimental and the cascading nature of\n                        // drop. This means that we consider an operator\n                        // unstable if either of its arguments or the operator\n                        // itself are in the experimental schema\n                        if operator.contains(\"toolkit_experimental.\") {\n                            return None;\n                        }\n\n                        if stable_operators.contains(operator) {\n                            return None;\n                        }\n\n                        return Some(val);\n                    }\n\n                    if let Some(cast) = val.strip_prefix(\"cast \") {\n                        // casts cannot be schema-qualified, so we rely on one\n                        // of the types involved being experimental and the\n                        // cascading nature of drop. This means that we consider\n                        // a cast unstable if and only if one of the types\n                        // involved are in the experimental schema\n                        if cast.contains(\"toolkit_experimental.\") {\n                            return None;\n                        }\n\n                        return Some(val);\n                    }\n\n                    Some(val)\n                })\n                .collect();\n\n            if unexpected_features.is_empty() {\n                return;\n            }\n\n            panic!(\"unexpectedly released features: {unexpected_features:#?}\")\n        });\n    }\n\n    fn stable_functions() -> HashSet<String> {\n        crate::stabilization_info::STABLE_FUNCTIONS()\n    }\n\n    fn stable_types() -> HashSet<String> {\n        crate::stabilization_info::STABLE_TYPES()\n    }\n\n    fn stable_operators() -> HashSet<String> {\n        crate::stabilization_info::STABLE_OPERATORS()\n    }\n}\n\n#[macro_export]\nmacro_rules! functions_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($fn_name: ident ( $( $($fn_type: ident)+ ),* ) ),* $(,)?\n            }\n        )*\n    ) => {\n        #[cfg(any(test, feature = \"pg_test\"))]\n        #[allow(non_snake_case)]\n        // we do this instead of just stringifying everything b/c stringify adds\n        // whitespace in places we don't want\n        pub fn $export_symbol() -> std::collections::HashSet<String> {\n            static FUNCTIONS: &[(&str, &[&str])] = &[\n                $(\n                    $(\n                        (\n                            stringify!($fn_name),\n                            &[\n                                $( stringify!($($fn_type)+) ),*\n                            ]\n                        ),\n                    )*\n                )*\n            ];\n            FUNCTIONS.iter().map(|(name, types)| {\n                format!(\"{}({})\", name, types.join(\",\"))\n            }).collect()\n        }\n\n    };\n}\n\n#[macro_export]\nmacro_rules! types_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($type_name: ident),* $(,)?\n            }\n        )*\n    ) => {\n        #[cfg(any(test, feature = \"pg_test\"))]\n        #[allow(non_snake_case)]\n        // we do this instead of just stringifying everything b/c stringify adds\n        // whitespace in places we don't want\n        pub fn $export_symbol() -> std::collections::HashSet<String> {\n            pub static TYPES: &[&str] = &[\n                $(\n                    $(stringify!($type_name),)*\n                )*\n            ];\n            TYPES.iter().map(|s| s.to_ascii_lowercase()).collect()\n        }\n    };\n}\n\n#[macro_export]\nmacro_rules! operators_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($operator_name: literal ( $( $($fn_type: ident)+ ),* ) ),* $(,)?\n            }\n        )*\n    ) => {\n        #[cfg(any(test, feature = \"pg_test\"))]\n        #[allow(non_snake_case)]\n        pub fn $export_symbol() -> std::collections::HashSet<String> {\n            static OPERATORS: &[(&str, &[&str])] = &[\n                $(\n                    $(\n                        (\n                            $operator_name,\n                            &[\n                                $( stringify!($($fn_type)+) ),*\n                            ]\n                        ),\n                    )*\n                )*\n            ];\n            OPERATORS.iter().map(|(name, types)| {\n                format!(\"{}({})\", name, types.join(\",\"))\n            }).collect()\n        }\n    };\n}\n"
  },
  {
    "path": "extension/src/state_aggregate/accessors.rs",
    "content": "use crate::{\n    datum_utils::interval_to_ms,\n    pg_type,\n    raw::{Interval, TimestampTz},\n    ron_inout_funcs,\n    state_aggregate::*,\n};\n\npg_type! {\n    struct AccessorInterpolatedStateTimeline<'input> {\n        start: i64,\n        interval: i64,\n        prev: StateAggData<'input>,\n        prev_present: bool,\n    }\n}\nron_inout_funcs!(AccessorInterpolatedStateTimeline<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_state_timeline\")]\nfn accessor_state_agg_interpolated_interpolated_state_timeline<'a>(\n    start: TimestampTz,\n    interval: Interval,\n    prev: Option<StateAgg<'a>>,\n) -> AccessorInterpolatedStateTimeline<'a> {\n    crate::build! {\n        AccessorInterpolatedStateTimeline {\n            interval: interval_to_ms(&start, &interval),\n            start: start.into(),\n            prev_present: prev.is_some(),\n            prev: prev.unwrap_or_else(|| StateAgg::empty(false)).0,\n        }\n    }\n}\n\npg_type! {\n    struct AccessorInterpolatedStateIntTimeline<'input> {\n        start: i64,\n        interval: i64,\n        prev: StateAggData<'input>,\n        prev_present: bool,\n    }\n}\nron_inout_funcs!(AccessorInterpolatedStateIntTimeline<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_state_int_timeline\")]\nfn accessor_state_agg_interpolated_interpolated_state_int_timeline<'a>(\n    start: TimestampTz,\n    interval: Interval,\n    prev: Option<StateAgg<'a>>,\n) -> AccessorInterpolatedStateIntTimeline<'a> {\n    crate::build! {\n        AccessorInterpolatedStateIntTimeline {\n            interval: interval_to_ms(&start, &interval),\n            start: start.into(),\n            prev_present: prev.is_some(),\n            prev: prev.unwrap_or_else(|| StateAgg::empty(false)).0,\n        }\n    }\n}\n\n// weird ordering is needed for alignment\npg_type! {\n    struct AccessorInterpolatedDurationIn<'input> {\n        start: i64,\n        interval: i64,\n        state_len: u32,\n        padding_2: [u8; 4],\n        prev: StateAggData<'input>,\n        state_bytes: [u8; self.state_len],\n        prev_present: bool,\n    }\n}\nron_inout_funcs!(AccessorInterpolatedDurationIn<'input>);\npg_type! {\n    struct AccessorInterpolatedDurationInInt<'input> {\n        start: i64,\n        interval: i64,\n        state: i64,\n        prev_present: bool,\n        padding_2: [u8; 7],\n        prev: StateAggData<'input>,\n    }\n}\nron_inout_funcs!(AccessorInterpolatedDurationInInt<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_duration_in\")]\nfn accessor_state_agg_interpolated_interpolated_duration_in<'a>(\n    state: String,\n    start: TimestampTz,\n    interval: Interval,\n    prev: Option<StateAgg<'a>>,\n) -> AccessorInterpolatedDurationIn<'a> {\n    crate::build! {\n        AccessorInterpolatedDurationIn {\n            state_len: state.len().try_into().unwrap(),\n            state_bytes: state.into_bytes().into(),\n            interval: interval_to_ms(&start, &interval),\n            start: start.into(),\n            prev_present: prev.is_some(),\n            prev: prev.unwrap_or_else(|| StateAgg::empty(false)).0,\n            padding_2: Default::default(),\n        }\n    }\n}\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_duration_in\")]\nfn accessor_state_agg_interpolated_interpolated_duration_in_int<'a>(\n    state: i64,\n    start: TimestampTz,\n    interval: Interval,\n    prev: Option<StateAgg<'a>>,\n) -> AccessorInterpolatedDurationInInt<'a> {\n    crate::build! {\n        AccessorInterpolatedDurationInInt {\n            state,\n            interval: interval_to_ms(&start, &interval),\n            start: start.into(),\n            prev_present: prev.is_some(),\n            prev: prev.unwrap_or_else(|| StateAgg::empty(false)).0,\n            padding_2: Default::default(),\n        }\n    }\n}\n\n// weird ordering is needed for alignment\npg_type! {\n    struct AccessorInterpolatedStatePeriods<'input> {\n        start: i64,\n        interval: i64,\n        state_len: u32,\n        padding_2: [u8; 4],\n        prev: StateAggData<'input>,\n        state_bytes: [u8; self.state_len],\n        prev_present: bool,\n    }\n}\nron_inout_funcs!(AccessorInterpolatedStatePeriods<'input>);\npg_type! {\n    struct AccessorInterpolatedStatePeriodsInt<'input> {\n        start: i64,\n        interval: i64,\n        state: i64,\n        prev_present: bool,\n        padding_2: [u8; 7],\n        prev: StateAggData<'input>,\n    }\n}\nron_inout_funcs!(AccessorInterpolatedStatePeriodsInt<'input>);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_state_periods\")]\nfn accessor_state_agg_interpolated_interpolated_state_periods<'a>(\n    state: String,\n    start: TimestampTz,\n    interval: Interval,\n    prev: Option<StateAgg<'a>>,\n) -> AccessorInterpolatedStatePeriods<'a> {\n    crate::build! {\n        AccessorInterpolatedStatePeriods {\n            state_len: state.len().try_into().unwrap(),\n            state_bytes: state.into_bytes().into(),\n            interval: interval_to_ms(&start, &interval),\n            start: start.into(),\n            prev_present: prev.is_some(),\n            prev: prev.unwrap_or_else(|| StateAgg::empty(false)).0,\n            padding_2: Default::default(),\n        }\n    }\n}\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_state_periods\")]\nfn accessor_state_agg_interpolated_interpolated_state_periods_int<'a>(\n    state: i64,\n    start: TimestampTz,\n    interval: Interval,\n    prev: Option<StateAgg<'a>>,\n) -> AccessorInterpolatedStatePeriodsInt<'a> {\n    crate::build! {\n        AccessorInterpolatedStatePeriodsInt {\n            state,\n            interval: interval_to_ms(&start, &interval),\n            start: start.into(),\n            prev_present: prev.is_some(),\n            prev: prev.unwrap_or_else(|| StateAgg::empty(false)).0,\n            padding_2: Default::default(),\n        }\n    }\n}\n\npg_type! {\n    struct AccessorDurationIn<'input> {\n        state_len: u32,\n        state_bytes: [u8; self.state_len],\n    }\n}\nron_inout_funcs!(AccessorDurationIn<'input>);\npg_type! {\n    struct AccessorDurationInInt {\n        state: i64,\n    }\n}\nron_inout_funcs!(AccessorDurationInInt);\n\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\nfn accessor_state_agg_duration_in(state: String) -> AccessorDurationIn<'static> {\n    crate::build! {\n        AccessorDurationIn {\n            state_len: state.len().try_into().unwrap(),\n            state_bytes: state.into_bytes().into(),\n        }\n    }\n}\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\nfn accessor_state_agg_duration_in_int(state: i64) -> AccessorDurationInInt {\n    crate::build! {\n        AccessorDurationInInt {\n            state,\n        }\n    }\n}\n\npg_type! {\n    struct AccessorStatePeriods<'input> {\n        state_len: u32,\n        state_bytes: [u8; self.state_len],\n    }\n}\nron_inout_funcs!(AccessorStatePeriods<'input>);\npg_type! {\n    struct AccessorStatePeriodsInt {\n        state: i64,\n    }\n}\nron_inout_funcs!(AccessorStatePeriodsInt);\n\n#[pg_extern(immutable, parallel_safe, name = \"state_periods\")]\nfn accessor_state_agg_state_periods<'a>(state: String) -> AccessorStatePeriods<'static> {\n    crate::build! {\n        AccessorStatePeriods {\n            state_len: state.len().try_into().unwrap(),\n            state_bytes: state.into_bytes().into(),\n        }\n    }\n}\n#[pg_extern(immutable, parallel_safe, name = \"state_periods\")]\nfn accessor_state_agg_state_periods_int(state: i64) -> AccessorStatePeriodsInt {\n    crate::build! {\n        AccessorStatePeriodsInt {\n            state,\n        }\n    }\n}\n\npg_type! {\n    struct AccessorDurationInRange<'input> {\n        state_len: u32,\n        padding_2: [u8; 4],\n        start: i64,\n        interval: i64,\n        state_bytes: [u8; self.state_len],\n    }\n}\nron_inout_funcs!(AccessorDurationInRange<'input>);\npg_type! {\n    struct AccessorDurationInRangeInt {\n        state: i64,\n        start: i64,\n        interval: i64,\n    }\n}\nron_inout_funcs!(AccessorDurationInRangeInt);\n\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\nfn accessor_state_agg_duration_in_range(\n    state: String,\n    start: TimestampTz,\n    interval: default!(Option<crate::raw::Interval>, \"NULL\"),\n) -> AccessorDurationInRange<'static> {\n    let interval = interval\n        .map(|interval| crate::datum_utils::interval_to_ms(&start, &interval))\n        .unwrap_or(NO_INTERVAL_MARKER);\n    let start = start.into();\n    crate::build! {\n        AccessorDurationInRange {\n            state_len: state.len().try_into().unwrap(),\n            state_bytes: state.into_bytes().into(),\n            padding_2: [0; 4],\n            start, interval\n        }\n    }\n}\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\nfn accessor_state_agg_duration_in_range_int(\n    state: i64,\n    start: TimestampTz,\n    interval: default!(Option<crate::raw::Interval>, \"NULL\"),\n) -> AccessorDurationInRangeInt {\n    let interval = interval\n        .map(|interval| crate::datum_utils::interval_to_ms(&start, &interval))\n        .unwrap_or(NO_INTERVAL_MARKER);\n    let start = start.into();\n    crate::build! {\n        AccessorDurationInRangeInt {\n            state,\n            start, interval\n        }\n    }\n}\n\npg_type! {\n    struct AccessorStateAt {\n        time: i64,\n    }\n}\nron_inout_funcs!(AccessorStateAt);\n\n#[pg_extern(immutable, parallel_safe, name = \"state_at\")]\nfn accessor_state_agg_state_at(time: TimestampTz) -> AccessorStateAt {\n    crate::build! {\n        AccessorStateAt {\n            time: time.into(),\n        }\n    }\n}\n\npg_type! {\n    struct AccessorStateAtInt {\n        time: i64,\n    }\n}\nron_inout_funcs!(AccessorStateAtInt);\n\n#[pg_extern(immutable, parallel_safe, name = \"state_at_int\")]\nfn accessor_state_agg_state_at_int(time: TimestampTz) -> AccessorStateAtInt {\n    crate::build! {\n        AccessorStateAtInt {\n            time: time.into(),\n        }\n    }\n}\n"
  },
  {
    "path": "extension/src/state_aggregate/rollup.rs",
    "content": "use super::{toolkit_experimental::*, *};\nuse crate::{\n    aggregate_utils::in_aggregate_context,\n    palloc::{InternalAsValue, ToInternal},\n};\nuse serde::{Deserialize, Serialize};\n\nextension_sql!(\n    \"CREATE AGGREGATE toolkit_experimental.rollup(\n        value toolkit_experimental.CompactStateAgg\n    ) (\n        sfunc = toolkit_experimental.compact_state_agg_rollup_trans,\n        stype = internal,\n        finalfunc = toolkit_experimental.compact_state_agg_rollup_final,\n        combinefunc = state_agg_rollup_combine,\n        serialfunc = state_agg_rollup_serialize,\n        deserialfunc = state_agg_rollup_deserialize,\n        parallel = restricted\n    );\",\n    name = \"compact_state_agg_rollup\",\n    requires = [\n        compact_state_agg_rollup_trans,\n        compact_state_agg_rollup_final,\n        state_agg_rollup_combine,\n        state_agg_rollup_serialize,\n        state_agg_rollup_deserialize,\n        CompactStateAgg,\n    ],\n);\nextension_sql!(\n    \"CREATE AGGREGATE rollup(\n        value StateAgg\n    ) (\n        sfunc = state_agg_rollup_trans,\n        stype = internal,\n        finalfunc = state_agg_rollup_final,\n        combinefunc = state_agg_rollup_combine,\n        serialfunc = state_agg_rollup_serialize,\n        deserialfunc = state_agg_rollup_deserialize,\n        parallel = restricted\n    );\",\n    name = \"state_agg_rollup\",\n    requires = [\n        state_agg_rollup_trans,\n        state_agg_rollup_final,\n        state_agg_rollup_combine,\n        state_agg_rollup_serialize,\n        state_agg_rollup_deserialize,\n        StateAgg,\n    ],\n);\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\npub struct RollupTransState {\n    values: Vec<OwnedCompactStateAgg>,\n    compact: bool,\n}\n\n#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]\nstruct OwnedCompactStateAgg {\n    durations: Vec<DurationInState>,\n    combined_durations: Vec<TimeInState>,\n    first_time: i64,\n    last_time: i64,\n    first_state: u32,\n    last_state: u32,\n    states: Vec<u8>,\n    compact: bool,\n    integer_states: bool,\n}\n\nimpl OwnedCompactStateAgg {\n    pub fn merge(self, other: Self) -> Self {\n        assert_eq!(\n            self.compact, other.compact,\n            \"can't merge compact_state_agg and state_agg\"\n        );\n        assert_eq!(\n            self.integer_states, other.integer_states,\n            \"can't merge aggs with different state types\"\n        );\n\n        let (earlier, later) = match self.cmp(&other) {\n            Ordering::Less => (self, other),\n            Ordering::Greater => (other, self),\n            Ordering::Equal => panic!(\n                \"can't merge overlapping aggregates (same start time: {})\",\n                self.first_time\n            ),\n        };\n\n        assert!(\n            earlier.last_time <= later.first_time,\n            \"can't merge overlapping aggregates (earlier={}-{}, later={}-{})\",\n            earlier.first_time,\n            earlier.last_time,\n            later.first_time,\n            later.last_time,\n        );\n        assert_ne!(\n            later.durations.len(),\n            0,\n            \"later aggregate must be non-empty\"\n        );\n        assert_ne!(\n            earlier.durations.len(),\n            0,\n            \"earlier aggregate must be non-empty\"\n        );\n\n        let later_states =\n            String::from_utf8(later.states.to_vec()).expect(\"invalid later UTF-8 states\");\n        let mut merged_states =\n            String::from_utf8(earlier.states.to_vec()).expect(\"invalid earlier UTF-8 states\");\n        let mut merged_durations = earlier.durations.into_iter().collect::<Vec<_>>();\n\n        let earlier_len = earlier.combined_durations.len();\n\n        let mut merged_last_state = None;\n        for (later_idx, dis) in later.durations.iter().enumerate() {\n            let materialized_dis = dis.state.materialize(&later_states);\n            let merged_duration_info =\n                merged_durations\n                    .iter_mut()\n                    .enumerate()\n                    .find(|(_, merged_dis)| {\n                        merged_dis.state.materialize(&merged_states) == materialized_dis\n                    });\n\n            let merged_idx =\n                if let Some((merged_idx, merged_duration_to_update)) = merged_duration_info {\n                    merged_duration_to_update.duration += dis.duration;\n                    merged_idx\n                } else {\n                    let state = materialized_dis.entry(&mut merged_states);\n                    merged_durations.push(DurationInState {\n                        state,\n                        duration: dis.duration,\n                    });\n                    merged_durations.len() - 1\n                };\n\n            if later_idx == later.last_state as usize {\n                // this is the last state\n                merged_last_state = Some(merged_idx);\n            };\n        }\n        let merged_last_state =\n            merged_last_state.expect(\"later last_state not in later.durations\") as u32;\n\n        let mut combined_durations = earlier\n            .combined_durations\n            .into_iter()\n            .chain(later.combined_durations.into_iter().map(|tis| {\n                let state = tis\n                    .state\n                    .materialize(&later_states)\n                    .existing_entry(&merged_states);\n                TimeInState { state, ..tis }\n            }))\n            .collect::<Vec<_>>();\n\n        let gap = later.first_time - earlier.last_time;\n        assert!(gap >= 0);\n        merged_durations\n            .get_mut(earlier.last_state as usize)\n            .expect(\"earlier.last_state doesn't point to a state\")\n            .duration += gap;\n\n        // ensure combined_durations covers the whole range of time\n        if !earlier.compact {\n            if combined_durations\n                .get_mut(earlier_len - 1)\n                .expect(\"invalid combined_durations: nothing at end of earlier\")\n                .state\n                .materialize(&merged_states)\n                == combined_durations\n                    .get(earlier_len)\n                    .expect(\"invalid combined_durations: nothing at start of earlier\")\n                    .state\n                    .materialize(&merged_states)\n            {\n                combined_durations\n                    .get_mut(earlier_len - 1)\n                    .expect(\"invalid combined_durations (nothing at earlier_len - 1, equal)\")\n                    .end_time = combined_durations.remove(earlier_len).end_time;\n            } else {\n                combined_durations\n                    .get_mut(earlier_len - 1)\n                    .expect(\"invalid combined_durations (nothing at earlier_len - 1, not equal)\")\n                    .end_time = combined_durations\n                    .get(earlier_len)\n                    .expect(\"invalid combined_durations (nothing at earlier_len, not equal)\")\n                    .start_time;\n            }\n        }\n\n        let merged_states = merged_states.into_bytes();\n        OwnedCompactStateAgg {\n            states: merged_states,\n            durations: merged_durations,\n            combined_durations,\n\n            first_time: earlier.first_time,\n            last_time: later.last_time,\n            first_state: earlier.first_state, // indexes into earlier durations are same for merged_durations\n            last_state: merged_last_state,\n\n            // these values are always the same for both\n            compact: earlier.compact,\n            integer_states: earlier.integer_states,\n        }\n    }\n}\n\nimpl<'a> From<OwnedCompactStateAgg> for CompactStateAgg<'a> {\n    fn from(owned: OwnedCompactStateAgg) -> CompactStateAgg<'a> {\n        unsafe {\n            flatten!(CompactStateAgg {\n                states_len: owned.states.len() as u64,\n                states: (&*owned.states).into(),\n                durations_len: owned.durations.len() as u64,\n                durations: (&*owned.durations).into(),\n                combined_durations: (&*owned.combined_durations).into(),\n                combined_durations_len: owned.combined_durations.len() as u64,\n                first_time: owned.first_time,\n                last_time: owned.last_time,\n                first_state: owned.first_state,\n                last_state: owned.last_state,\n                compact: owned.compact,\n                integer_states: owned.integer_states,\n            })\n        }\n    }\n}\n\nimpl<'a> From<CompactStateAgg<'a>> for OwnedCompactStateAgg {\n    fn from(agg: CompactStateAgg<'a>) -> OwnedCompactStateAgg {\n        OwnedCompactStateAgg {\n            states: agg.states.iter().collect::<Vec<_>>(),\n            durations: agg.durations.iter().collect::<Vec<_>>(),\n            combined_durations: agg.combined_durations.iter().collect::<Vec<_>>(),\n            first_time: agg.first_time,\n            last_time: agg.last_time,\n            first_state: agg.first_state,\n            last_state: agg.last_state,\n            compact: agg.compact,\n            integer_states: agg.integer_states,\n        }\n    }\n}\n\nimpl PartialOrd for OwnedCompactStateAgg {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Ord for OwnedCompactStateAgg {\n    fn cmp(&self, other: &Self) -> Ordering {\n        // compare using first time (OwnedCompactStateAgg::merge will handle any overlap)\n        self.first_time.cmp(&other.first_time)\n    }\n}\n\nimpl RollupTransState {\n    fn merge(&mut self) {\n        // OwnedCompactStateAgg::merge can't merge overlapping aggregates\n        self.values.sort();\n        self.values = self\n            .values\n            .drain(..)\n            .reduce(|a, b| a.merge(b))\n            .map(|val| vec![val])\n            .unwrap_or_else(Vec::new);\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn compact_state_agg_rollup_trans(\n    state: Internal,\n    next: Option<CompactStateAgg>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    compact_state_agg_rollup_trans_inner(unsafe { state.to_inner() }, next, fcinfo).internal()\n}\n\npub fn compact_state_agg_rollup_trans_inner(\n    state: Option<Inner<RollupTransState>>,\n    next: Option<CompactStateAgg>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<RollupTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, next) {\n            (None, None) => None,\n            (None, Some(next)) => Some(\n                RollupTransState {\n                    values: vec![next.into()],\n                    compact: false,\n                }\n                .into(),\n            ),\n            (Some(state), None) => Some(state),\n            (Some(mut state), Some(next)) => {\n                state.values.push(next.into());\n                Some(state)\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn state_agg_rollup_trans(\n    state: Internal,\n    next: Option<StateAgg>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    compact_state_agg_rollup_trans_inner(\n        unsafe { state.to_inner() },\n        next.map(StateAgg::as_compact_state_agg),\n        fcinfo,\n    )\n    .internal()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn compact_state_agg_rollup_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<CompactStateAgg<'static>> {\n    compact_state_agg_rollup_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\n\nfn compact_state_agg_rollup_final_inner(\n    state: Option<Inner<RollupTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<CompactStateAgg<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state.clone(),\n            };\n            state.merge();\n            assert!(state.values.len() == 1);\n            let agg: Option<OwnedCompactStateAgg> = state.values.drain(..).next().unwrap().into();\n            agg.map(Into::into)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn state_agg_rollup_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<StateAgg<'static>> {\n    state_agg_rollup_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\n\nfn state_agg_rollup_final_inner(\n    state: Option<Inner<RollupTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<StateAgg<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state.clone(),\n            };\n            state.merge();\n            assert!(state.values.len() == 1);\n            let agg: Option<OwnedCompactStateAgg> = state.values.drain(..).next().unwrap().into();\n            agg.map(Into::into).map(StateAgg::new)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn state_agg_rollup_serialize(state: Internal) -> bytea {\n    let mut state: Inner<RollupTransState> = unsafe { state.to_inner().unwrap() };\n    state.merge();\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn state_agg_rollup_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    state_agg_rollup_deserialize_inner(bytes).internal()\n}\npub fn state_agg_rollup_deserialize_inner(bytes: bytea) -> Inner<RollupTransState> {\n    let t: RollupTransState = crate::do_deserialize!(bytes, RollupTransState);\n    t.into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn state_agg_rollup_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe {\n        state_agg_rollup_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal()\n    }\n}\n\n#[allow(clippy::redundant_clone)] // clone is needed so we don't mutate shared memory\npub fn state_agg_rollup_combine_inner(\n    state1: Option<Inner<RollupTransState>>,\n    state2: Option<Inner<RollupTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<RollupTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (Some(x), None) => Some(x.clone().into()),\n            (None, Some(x)) => Some(x.clone().into()),\n            (Some(x), Some(y)) => {\n                let compact = x.compact;\n                assert_eq!(\n                    compact, y.compact,\n                    \"trying to merge compact and non-compact state aggs, this should be unreachable\"\n                );\n                let values = x\n                    .values\n                    .iter()\n                    .chain(y.values.iter())\n                    .map(Clone::clone)\n                    .collect::<Vec<_>>();\n                let trans_state = RollupTransState { values, compact };\n                Some(trans_state.clone().into())\n            }\n        })\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    #[should_panic = \"can't merge overlapping aggregates\"]\n    fn merge_range_full_overlap() {\n        let mut outer: OwnedCompactStateAgg = CompactStateAgg::empty(false, false).into();\n        outer.first_time = 10;\n        outer.last_time = 50;\n\n        let mut inner: OwnedCompactStateAgg = CompactStateAgg::empty(false, false).into();\n        inner.first_time = 20;\n        inner.last_time = 30;\n\n        inner.merge(outer);\n    }\n\n    #[pg_test]\n    #[should_panic = \"can't merge overlapping aggregates\"]\n    fn merge_range_partial_overlap() {\n        let mut r1: OwnedCompactStateAgg = CompactStateAgg::empty(false, false).into();\n        r1.first_time = 10;\n        r1.last_time = 50;\n\n        let mut r2: OwnedCompactStateAgg = CompactStateAgg::empty(false, false).into();\n        r2.first_time = 20;\n        r2.last_time = 50;\n\n        r2.merge(r1);\n    }\n\n    #[test]\n    fn merges_compact_aggs_correctly() {\n        let s1 = OwnedCompactStateAgg {\n            durations: vec![\n                DurationInState {\n                    duration: 500,\n                    state: StateEntry::from_integer(5_552),\n                },\n                DurationInState {\n                    duration: 400,\n                    state: StateEntry::from_integer(5_551),\n                },\n            ],\n            combined_durations: vec![],\n            first_time: 100,\n            last_time: 1000,\n            first_state: 1,\n            last_state: 0,\n            states: vec![],\n            compact: true,\n            integer_states: true,\n        };\n        let s2 = OwnedCompactStateAgg {\n            durations: vec![\n                DurationInState {\n                    duration: 500,\n                    state: StateEntry::from_integer(5_552),\n                },\n                DurationInState {\n                    duration: 400,\n                    state: StateEntry::from_integer(5_551),\n                },\n            ],\n            combined_durations: vec![],\n            first_time: 1000 + 12345,\n            last_time: 1900 + 12345,\n            first_state: 1,\n            last_state: 0,\n            states: vec![],\n            compact: true,\n            integer_states: true,\n        };\n        let s3 = OwnedCompactStateAgg {\n            durations: vec![\n                DurationInState {\n                    duration: 500,\n                    state: StateEntry::from_integer(5_552),\n                },\n                DurationInState {\n                    duration: 400,\n                    state: StateEntry::from_integer(5_551),\n                },\n            ],\n            combined_durations: vec![],\n            first_time: 1900 + 12345,\n            last_time: 1900 + 12345 + 900,\n            first_state: 1,\n            last_state: 0,\n            states: vec![],\n            compact: true,\n            integer_states: true,\n        };\n        let expected = OwnedCompactStateAgg {\n            durations: vec![\n                DurationInState {\n                    duration: 500 * 3 + 12345,\n                    state: StateEntry::from_integer(5_552),\n                },\n                DurationInState {\n                    duration: 400 * 3,\n                    state: StateEntry::from_integer(5_551),\n                },\n            ],\n            combined_durations: vec![],\n            first_time: 100,\n            last_time: 1900 + 12345 + 900,\n            first_state: 1,\n            last_state: 0,\n            states: vec![],\n            compact: true,\n            integer_states: true,\n        };\n        let merged = s1.clone().merge(s2.clone().merge(s3.clone()));\n        assert_eq!(merged, expected);\n        let merged = s3.clone().merge(s2.clone().merge(s1.clone()));\n        assert_eq!(merged, expected);\n\n        let mut trans_state = RollupTransState {\n            values: vec![s1.clone(), s2.clone(), s3.clone()],\n            compact: true,\n        };\n        trans_state.merge();\n        assert_eq!(trans_state.values.len(), 1);\n        assert_eq!(trans_state.values[0], expected.clone());\n\n        let mut trans_state = RollupTransState {\n            values: vec![s3.clone(), s1.clone(), s2.clone()],\n            compact: true,\n        };\n        trans_state.merge();\n        assert_eq!(trans_state.values.len(), 1);\n        assert_eq!(trans_state.values[0], expected.clone());\n    }\n}\n"
  },
  {
    "path": "extension/src/state_aggregate.rs",
    "content": "//! SELECT duration_in('STOPPED', states) as run_time, duration_in('ERROR', states) as error_time FROM (\n//!   SELECT compact_state_agg(time, state) as states FROM ...\n//! );\n//!\n//! Currently requires loading all data into memory in order to sort it by time.\n\n#![allow(non_camel_case_types)]\n\nuse pgrx::{iter::TableIterator, *};\nuse serde::{Deserialize, Serialize};\nuse std::cmp::Ordering;\n\nuse aggregate_builder::aggregate;\nuse flat_serialize::*;\nuse flat_serialize_macro::FlatSerializable;\n\nuse crate::{\n    accessors::{\n        AccessorIntoIntValues, AccessorIntoValues, AccessorStateIntTimeline, AccessorStateTimeline,\n    },\n    flatten,\n    palloc::{Inner, Internal},\n    pg_type,\n    raw::{bytea, TimestampTz},\n    ron_inout_funcs,\n};\n\nuse toolkit_experimental::{CompactStateAgg, CompactStateAggData};\n\nmod accessors;\nuse accessors::*;\npub mod rollup;\n\n/// The data of a state.\n#[derive(Debug, Clone, PartialEq, Eq, Ord, PartialOrd, Hash, Serialize, Deserialize)]\n#[repr(C)]\nenum MaterializedState {\n    String(String),\n    Integer(i64),\n}\nimpl MaterializedState {\n    fn entry(&self, states: &mut String) -> StateEntry {\n        match self {\n            Self::Integer(i) => StateEntry { a: i64::MAX, b: *i },\n            Self::String(s) => StateEntry::from_str(states, s),\n        }\n    }\n    fn existing_entry(&self, states: &str) -> StateEntry {\n        match self {\n            Self::Integer(i) => StateEntry { a: i64::MAX, b: *i },\n            Self::String(s) => StateEntry::from_existing_str(states, s),\n        }\n    }\n\n    fn into_string(self) -> String {\n        match self {\n            Self::String(str) => str,\n            _ => panic!(\"MaterializedState::into_string called with non-string\"),\n        }\n    }\n    fn into_integer(self) -> i64 {\n        match self {\n            Self::Integer(int) => int,\n            _ => panic!(\"MaterializedState::into_integer called with non-integer\"),\n        }\n    }\n}\n\n/// A stored state entry. Needs a `states` string to be interpreted.\n#[derive(\n    Debug, Copy, Clone, PartialEq, Eq, Ord, PartialOrd, FlatSerializable, Serialize, Deserialize,\n)]\n#[repr(C)]\npub struct StateEntry {\n    a: i64,\n    b: i64,\n}\nimpl StateEntry {\n    #[cfg(test)] // only used by tests\n    fn from_integer(int: i64) -> Self {\n        Self {\n            a: i64::MAX,\n            b: int,\n        }\n    }\n    fn from_str(states: &mut String, new_state: &str) -> Self {\n        let (a, b) = if let Some(bounds) = states\n            .find(new_state)\n            .map(|idx| (idx as i64, (idx + new_state.len()) as i64))\n        {\n            bounds\n        } else {\n            let bounds = (states.len() as i64, (states.len() + new_state.len()) as i64);\n            states.push_str(new_state);\n            bounds\n        };\n        Self { a, b }\n    }\n    fn from_existing_str(states: &str, state: &str) -> Self {\n        if let Some(val) = Self::try_from_existing_str(states, state) {\n            val\n        } else {\n            panic!(\"Tried to get state that doesn't exist: {state}\")\n        }\n    }\n    fn try_from_existing_str(states: &str, state: &str) -> Option<Self> {\n        states\n            .find(state)\n            .map(|idx| (idx as i64, (idx + state.len()) as i64))\n            .map(|bounds| Self {\n                a: bounds.0,\n                b: bounds.1,\n            })\n    }\n\n    fn materialize(&self, states: &str) -> MaterializedState {\n        if self.a == i64::MAX {\n            MaterializedState::Integer(self.b)\n        } else {\n            MaterializedState::String(\n                states\n                    .get(self.a as usize..self.b as usize)\n                    .expect(\"tried to materialize out-of-bounds state\")\n                    .to_string(),\n            )\n        }\n    }\n\n    fn as_str(self, states: &str) -> &str {\n        assert!(self.a != i64::MAX, \"Tried to get non-string state\");\n        states\n            .get(self.a as usize..self.b as usize)\n            .expect(\"tried to stringify out-of-bounds state\")\n    }\n\n    fn into_integer(self) -> i64 {\n        assert!(self.a == i64::MAX, \"Tried to get non-integer state\");\n        self.b\n    }\n}\n\n#[pg_schema]\npub mod toolkit_experimental {\n    use super::*;\n\n    pg_type! {\n        #[derive(Debug)]\n        struct CompactStateAgg<'input> {\n            states_len: u64, // TODO JOSH this and durations_len can be 32\n            durations_len: u64,\n            durations: [DurationInState; self.durations_len],\n            combined_durations_len: u64,\n            combined_durations: [TimeInState; self.combined_durations_len],\n            first_time: i64,\n            last_time: i64,\n            first_state: u32,\n            last_state: u32,  // first/last state are idx into durations, keep together for alignment\n            states: [u8; self.states_len],\n            compact: bool,\n            integer_states: bool,\n        }\n    }\n\n    impl CompactStateAgg<'_> {\n        pub(super) fn empty(compact: bool, integer_states: bool) -> Self {\n            unsafe {\n                flatten!(CompactStateAgg {\n                    states_len: 0,\n                    states: Slice::Slice(&[]),\n                    durations_len: 0,\n                    durations: Slice::Slice(&[]),\n                    combined_durations: Slice::Slice(&[]),\n                    combined_durations_len: 0,\n                    first_time: 0,\n                    last_time: 0,\n                    first_state: 0,\n                    last_state: 0,\n                    compact,\n                    integer_states,\n                })\n            }\n        }\n\n        pub(super) fn new(\n            states: String,\n            durations: Vec<DurationInState>,\n            first: Option<Record>,\n            last: Option<Record>,\n            combined_durations: Option<Vec<TimeInState>>,\n            integer_states: bool,\n        ) -> Self {\n            let compact = combined_durations.is_none();\n            if durations.is_empty() {\n                assert!(\n                    first.is_none()\n                        && last.is_none()\n                        && states.is_empty()\n                        && combined_durations.map(|v| v.is_empty()).unwrap_or(true)\n                );\n\n                return Self::empty(compact, integer_states);\n            }\n\n            assert!(first.is_some() && last.is_some());\n            let first = first.unwrap();\n            let last = last.unwrap();\n            let states_len = states.len() as u64;\n            let durations_len = durations.len() as u64;\n            let mut first_state = durations.len();\n            let mut last_state = durations.len();\n\n            // Find first and last state\n            for (i, d) in durations.iter().enumerate() {\n                let s = d.state.materialize(&states);\n                if s == first.state {\n                    first_state = i;\n                    if last_state < durations.len() {\n                        break;\n                    }\n                }\n                if s == last.state {\n                    last_state = i;\n                    if first_state < durations.len() {\n                        break;\n                    }\n                }\n            }\n            assert!(first_state < durations.len() && last_state < durations.len());\n\n            let combined_durations = combined_durations.unwrap_or_default();\n\n            unsafe {\n                flatten!(CompactStateAgg {\n                    states_len,\n                    states: states.into_bytes().into(),\n                    durations_len,\n                    durations: (&*durations).into(),\n                    combined_durations: (&*combined_durations).into(),\n                    combined_durations_len: combined_durations.len() as u64,\n                    first_time: first.time,\n                    last_time: last.time,\n                    first_state: first_state as u32,\n                    last_state: last_state as u32,\n                    compact,\n                    integer_states,\n                })\n            }\n        }\n\n        pub fn get(&self, state: StateEntry) -> Option<i64> {\n            self.get_materialized(&state.materialize(self.states_as_str()))\n        }\n        pub(super) fn get_materialized(&self, state: &MaterializedState) -> Option<i64> {\n            for record in self.durations.iter() {\n                if record.state.materialize(self.states_as_str()) == *state {\n                    return Some(record.duration);\n                }\n            }\n            None\n        }\n\n        pub(super) fn states_as_str(&self) -> &str {\n            let states: &[u8] = self.states.as_slice();\n            // SAFETY: came from a String in `new` a few lines up\n            unsafe { std::str::from_utf8_unchecked(states) }\n        }\n\n        pub(super) fn interpolate(\n            &self,\n            interval_start: i64,\n            interval_len: i64,\n            prev: Option<CompactStateAgg>,\n        ) -> CompactStateAgg<'_> {\n            if self.durations.is_empty() {\n                pgrx::error!(\"unable to interpolate interval on state aggregate with no data\");\n            }\n            if let Some(ref prev) = prev {\n                assert_eq!(\n                    prev.integer_states, self.integer_states,\n                    \"can't interpolate between aggs with different state types\"\n                );\n            }\n\n            let mut states = std::str::from_utf8(self.states.as_slice())\n                .unwrap()\n                .to_string();\n            let mut durations: Vec<DurationInState> = self.durations.iter().collect();\n\n            let mut combined_durations = if self.compact {\n                None\n            } else {\n                Some(self.combined_durations.iter().collect::<Vec<_>>())\n            };\n\n            let first = match prev {\n                Some(prev) if interval_start < self.first_time => {\n                    if prev.last_state < prev.durations.len() as u32 {\n                        let start_interval = self.first_time - interval_start;\n                        let start_state = &prev.durations.as_slice()[prev.last_state as usize]\n                            .state\n                            .materialize(prev.states_as_str());\n\n                        // update durations\n                        let state = match durations\n                            .iter_mut()\n                            .find(|x| x.state.materialize(&states) == *start_state)\n                        {\n                            Some(dis) => {\n                                dis.duration += start_interval;\n                                dis.state\n                            }\n                            None => {\n                                let state = start_state.entry(&mut states);\n                                durations.push(DurationInState {\n                                    duration: start_interval,\n                                    state,\n                                });\n                                state\n                            }\n                        };\n\n                        // update combined_durations\n                        if let Some(combined_durations) = combined_durations.as_mut() {\n                            // extend last duration\n                            let first_cd = combined_durations\n                                .first_mut()\n                                .expect(\"poorly formed StateAgg, length mismatch\");\n                            let first_cd_state = first_cd.state.materialize(&states);\n                            if first_cd_state == *start_state {\n                                first_cd.start_time -= start_interval;\n                            } else {\n                                combined_durations.insert(\n                                    0,\n                                    TimeInState {\n                                        start_time: interval_start,\n                                        end_time: self.first_time,\n                                        state,\n                                    },\n                                );\n                            };\n                        };\n\n                        Record {\n                            state: start_state.clone(),\n                            time: interval_start,\n                        }\n                    } else {\n                        pgrx::error!(\"unable to interpolate interval on state aggregate where previous agg has no data\")\n                    }\n                }\n                _ => Record {\n                    state: self.durations.as_slice()[self.first_state as usize]\n                        .state\n                        .materialize(&states),\n                    time: self.first_time,\n                },\n            };\n\n            let last = if interval_start + interval_len > self.last_time {\n                let last_interval = interval_start + interval_len - self.last_time;\n                match durations.get_mut(self.last_state as usize) {\n                    None => {\n                        pgrx::error!(\"poorly formed state aggregate, last_state out of starts\")\n                    }\n                    Some(dis) => {\n                        dis.duration += last_interval;\n                        if let Some(combined_durations) = combined_durations.as_mut() {\n                            // extend last duration\n                            combined_durations\n                                .last_mut()\n                                .expect(\"poorly formed state aggregate, length mismatch\")\n                                .end_time += last_interval;\n                        };\n                        Record {\n                            state: dis.state.materialize(&states),\n                            time: interval_start + interval_len,\n                        }\n                    }\n                }\n            } else {\n                Record {\n                    state: self.durations.as_slice()[self.last_state as usize]\n                        .state\n                        .materialize(&states),\n                    time: self.last_time,\n                }\n            };\n\n            CompactStateAgg::new(\n                states,\n                durations,\n                Some(first),\n                Some(last),\n                combined_durations,\n                self.integer_states,\n            )\n        }\n\n        pub fn assert_int<'a>(&self) {\n            assert!(\n                self.0.integer_states,\n                \"Expected integer state, found string state\"\n            );\n        }\n        pub fn assert_str<'a>(&self) {\n            assert!(\n                !self.0.integer_states,\n                \"Expected string state, found integer state\"\n            );\n        }\n    }\n\n    ron_inout_funcs!(CompactStateAgg<'input>);\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct StateAgg<'input> {\n        compact_state_agg: CompactStateAggData<'input>,\n    }\n}\nimpl<'input> StateAgg<'input> {\n    pub fn new(compact_state_agg: CompactStateAgg) -> Self {\n        unsafe {\n            flatten!(StateAgg {\n                compact_state_agg: compact_state_agg.0,\n            })\n        }\n    }\n\n    pub fn empty(integer_states: bool) -> Self {\n        Self::new(CompactStateAgg::empty(false, integer_states))\n    }\n\n    pub fn as_compact_state_agg(self) -> toolkit_experimental::CompactStateAgg<'static> {\n        unsafe { self.0.compact_state_agg.flatten() }\n    }\n\n    pub fn assert_int<'a>(&self) {\n        assert!(\n            self.0.compact_state_agg.integer_states,\n            \"State must have integer values for this function\"\n        );\n    }\n    pub fn assert_str<'a>(&self) {\n        assert!(\n            !self.0.compact_state_agg.integer_states,\n            \"State must have string values for this function\"\n        );\n    }\n}\nron_inout_funcs!(StateAgg<'input>);\n\nfn state_trans_inner(\n    state: Option<CompactStateAggTransState>,\n    ts: TimestampTz,\n    value: Option<MaterializedState>,\n    integer_states: bool,\n) -> Option<CompactStateAggTransState> {\n    let value = match value {\n        None => return state,\n        Some(value) => value,\n    };\n    let mut state = state.unwrap_or_else(|| CompactStateAggTransState::new(integer_states));\n    state.record(value, ts.into());\n    Some(state)\n}\n#[aggregate]\nimpl toolkit_experimental::compact_state_agg {\n    type State = CompactStateAggTransState;\n\n    const PARALLEL_SAFE: bool = true;\n\n    fn transition(\n        state: Option<State>,\n        #[sql_type(\"timestamptz\")] ts: TimestampTz,\n        #[sql_type(\"text\")] value: Option<String>,\n    ) -> Option<State> {\n        state_trans_inner(state, ts, value.map(MaterializedState::String), false)\n    }\n\n    fn combine(a: Option<&State>, b: Option<&State>) -> Option<State> {\n        match (a, b) {\n            (None, None) => None,\n            (None, Some(only)) | (Some(only), None) => Some(only.clone()),\n            (Some(a), Some(b)) => {\n                let (mut a, mut b) = (a.clone(), b.clone());\n                a.append(&mut b);\n                Some(a)\n            }\n        }\n    }\n\n    fn serialize(state: &mut State) -> bytea {\n        crate::do_serialize!(state)\n    }\n\n    fn deserialize(bytes: bytea) -> State {\n        crate::do_deserialize!(bytes, CompactStateAggTransState)\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<CompactStateAgg<'static>> {\n        state.map(|s| {\n            let mut states = String::new();\n            let mut durations: Vec<DurationInState> = vec![];\n            let (map, first, last) = s.make_duration_map_and_bounds();\n            for (state, duration) in map {\n                durations.push(DurationInState {\n                    duration,\n                    state: state.entry(&mut states),\n                });\n            }\n            CompactStateAgg::new(states, durations, first, last, None, s.integer_states)\n        })\n    }\n}\n\nextension_sql!(\n    \"CREATE AGGREGATE toolkit_experimental.compact_state_agg(\n        ts timestamptz,\n        value bigint\n    ) (\n        stype = internal,\n        sfunc = toolkit_experimental.compact_state_agg_int_trans,\n        finalfunc = toolkit_experimental.compact_state_agg_finally_fn_outer,\n        parallel = safe,\n        serialfunc = toolkit_experimental.compact_state_agg_serialize_fn_outer,\n        deserialfunc = toolkit_experimental.compact_state_agg_deserialize_fn_outer,\n        combinefunc = toolkit_experimental.compact_state_agg_combine_fn_outer\n    );\",\n    name = \"compact_state_agg_bigint\",\n    requires = [\n        compact_state_agg_int_trans,\n        compact_state_agg_finally_fn_outer,\n        compact_state_agg_serialize_fn_outer,\n        compact_state_agg_deserialize_fn_outer,\n        compact_state_agg_combine_fn_outer\n    ],\n);\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\nfn compact_state_agg_int_trans(\n    __inner: pgrx::Internal,\n    ts: TimestampTz,\n    value: Option<i64>,\n    __fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<pgrx::Internal> {\n    // expanded from #[aggregate] transition function\n    use crate::palloc::{Inner, InternalAsValue, ToInternal};\n    type State = CompactStateAggTransState;\n    unsafe {\n        let mut __inner: Option<Inner<Option<State>>> = __inner.to_inner();\n        let inner: Option<State> = match &mut __inner {\n            None => None,\n            Some(inner) => Option::take(&mut **inner),\n        };\n        let state: Option<State> = inner;\n        crate::aggregate_utils::in_aggregate_context(__fcinfo, || {\n            let result = state_trans_inner(state, ts, value.map(MaterializedState::Integer), true);\n            let state: Option<State> = result;\n            __inner = match (__inner, state) {\n                (None, None) => None,\n                (None, state @ Some(..)) => Some(state.into()),\n                (Some(mut inner), state) => {\n                    *inner = state;\n                    Some(inner)\n                }\n            };\n            __inner.internal()\n        })\n    }\n}\n\n#[aggregate]\nimpl state_agg {\n    type State = CompactStateAggTransState;\n\n    const PARALLEL_SAFE: bool = true;\n\n    fn transition(\n        state: Option<State>,\n        #[sql_type(\"timestamptz\")] ts: TimestampTz,\n        #[sql_type(\"text\")] value: Option<String>,\n    ) -> Option<State> {\n        compact_state_agg::transition(state, ts, value)\n    }\n\n    fn combine(a: Option<&State>, b: Option<&State>) -> Option<State> {\n        compact_state_agg::combine(a, b)\n    }\n\n    fn serialize(state: &mut State) -> bytea {\n        compact_state_agg::serialize(state)\n    }\n\n    fn deserialize(bytes: bytea) -> State {\n        compact_state_agg::deserialize(bytes)\n    }\n\n    fn finally(state: Option<&mut State>) -> Option<StateAgg<'static>> {\n        state.map(|s| {\n            let mut states = String::new();\n            let mut durations: Vec<DurationInState> = vec![];\n            let (map, first, last) = s.make_duration_map_and_bounds();\n            for (state, duration) in map {\n                let state = state.entry(&mut states);\n                durations.push(DurationInState { duration, state });\n            }\n\n            let mut merged_durations: Vec<TimeInState> = Vec::new();\n            let mut last_record_state = None;\n            for record in s.records.drain(..) {\n                if last_record_state\n                    .clone()\n                    .map(|last| last != record.state)\n                    .unwrap_or(true)\n                {\n                    if let Some(prev) = merged_durations.last_mut() {\n                        prev.end_time = record.time;\n                    }\n                    merged_durations.push(TimeInState {\n                        start_time: record.time,\n                        end_time: 0,\n                        state: record.state.entry(&mut states),\n                    });\n                    last_record_state = Some(record.state);\n                }\n            }\n            if let Some(last_time_in_state) = merged_durations.last_mut() {\n                last_time_in_state.end_time = last.as_ref().unwrap().time;\n            }\n\n            StateAgg::new(CompactStateAgg::new(\n                states,\n                durations,\n                first,\n                last,\n                Some(merged_durations),\n                s.integer_states,\n            ))\n        })\n    }\n}\n\nextension_sql!(\n    \"CREATE AGGREGATE state_agg(\n        ts timestamptz,\n        value bigint\n    ) (\n        stype = internal,\n        sfunc = state_agg_int_trans,\n        finalfunc = state_agg_finally_fn_outer,\n        parallel = safe,\n        serialfunc = state_agg_serialize_fn_outer,\n        deserialfunc = state_agg_deserialize_fn_outer,\n        combinefunc = state_agg_combine_fn_outer\n    );\",\n    name = \"state_agg_bigint\",\n    requires = [\n        state_agg_int_trans,\n        state_agg_finally_fn_outer,\n        state_agg_serialize_fn_outer,\n        state_agg_deserialize_fn_outer,\n        state_agg_combine_fn_outer\n    ],\n);\n#[pg_extern(immutable, parallel_safe)]\nfn state_agg_int_trans(\n    __inner: pgrx::Internal,\n    ts: TimestampTz,\n    value: Option<i64>,\n    __fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<pgrx::Internal> {\n    // expanded from #[aggregate] transition function\n    use crate::palloc::{Inner, InternalAsValue, ToInternal};\n    type State = CompactStateAggTransState;\n    unsafe {\n        let mut __inner: Option<Inner<Option<State>>> = __inner.to_inner();\n        let inner: Option<State> = match &mut __inner {\n            None => None,\n            Some(inner) => Option::take(&mut **inner),\n        };\n        let state: Option<State> = inner;\n        crate::aggregate_utils::in_aggregate_context(__fcinfo, || {\n            let result = state_trans_inner(state, ts, value.map(MaterializedState::Integer), true);\n            let state: Option<State> = result;\n            __inner = match (__inner, state) {\n                (None, None) => None,\n                (None, state @ Some(..)) => Some(state.into()),\n                (Some(mut inner), state) => {\n                    *inner = state;\n                    Some(inner)\n                }\n            };\n            __inner.internal()\n        })\n    }\n}\n\n// Intermediate state kept in postgres.\n#[derive(Serialize, Deserialize, Clone, Debug, PartialEq, Eq)]\npub struct CompactStateAggTransState {\n    records: Vec<Record>,\n    integer_states: bool,\n}\n\nimpl CompactStateAggTransState {\n    fn new(integer_states: bool) -> Self {\n        Self {\n            records: vec![],\n            integer_states,\n        }\n    }\n\n    fn record(&mut self, state: MaterializedState, time: i64) {\n        self.records.push(Record { state, time });\n    }\n\n    fn append(&mut self, other: &mut Self) {\n        self.records.append(&mut other.records)\n    }\n\n    fn sort_records(&mut self) {\n        self.records.sort_by(|a, b| {\n            if a.time == b.time {\n                // TODO JOSH do we care about instantaneous state changes?\n                //           an alternative is to drop duplicate timestamps\n                if a.state != b.state {\n                    // TODO use human-readable timestamp\n                    panic!(\n                        \"state cannot be both {:?} and {:?} at {}\",\n                        a.state, b.state, a.time\n                    )\n                }\n                std::cmp::Ordering::Equal\n            } else {\n                a.time.cmp(&b.time)\n            }\n        });\n    }\n\n    /// Use accumulated state, sort, and return tuple of map of states to durations along with first and last record.\n    fn make_duration_map_and_bounds(\n        &mut self,\n    ) -> (\n        std::collections::HashMap<MaterializedState, i64>,\n        Option<Record>,\n        Option<Record>,\n    ) {\n        self.sort_records();\n        let (first, last) = (self.records.first(), self.records.last());\n        let first = first.cloned();\n        let last = last.cloned();\n        let mut duration_state = DurationState::new();\n        for record in &self.records {\n            duration_state.handle_record(record.state.clone(), record.time);\n        }\n        duration_state.finalize();\n        // TODO BRIAN sort this by decreasing duration will make it easier to implement a TopN states\n        (duration_state.durations, first, last)\n    }\n}\n\nfn duration_in_inner<'a>(\n    aggregate: Option<CompactStateAgg<'a>>,\n    state: MaterializedState,\n    range: Option<(i64, Option<i64>)>, // start and interval\n) -> crate::raw::Interval {\n    let time: i64 = if let Some((start, interval)) = range {\n        let end = if let Some(interval) = interval {\n            assert!(interval >= 0, \"Interval must not be negative\");\n            start + interval\n        } else {\n            i64::MAX\n        };\n        assert!(end >= start, \"End time must be after start time\");\n        if let Some(agg) = aggregate {\n            assert!(\n                !agg.0.compact,\n                \"unreachable: interval specified for compact aggregate\"\n            );\n\n            let mut total = 0;\n            for tis in agg.combined_durations.iter() {\n                let tis_start_time = i64::max(tis.start_time, start);\n                let tis_end_time = i64::min(tis.end_time, end);\n                if tis_start_time > end {\n                    // combined_durations is sorted, so after this point there can't be any more\n                    break;\n                };\n                if tis_end_time >= start && tis.state.materialize(agg.states_as_str()) == state {\n                    let amount = tis_end_time - tis_start_time;\n                    assert!(amount >= 0, \"incorrectly ordered times\");\n                    total += amount;\n                }\n            }\n            total\n        } else {\n            0\n        }\n    } else {\n        aggregate\n            .and_then(|aggregate| aggregate.get_materialized(&state))\n            .unwrap_or(0)\n    };\n    time.into()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn duration_in<'a>(agg: Option<CompactStateAgg<'a>>, state: String) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    let state = MaterializedState::String(state);\n    duration_in_inner(agg, state, None)\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"duration_in\",\n    schema = \"toolkit_experimental\"\n)]\npub fn duration_in_int<'a>(agg: Option<CompactStateAgg<'a>>, state: i64) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    duration_in_inner(agg, MaterializedState::Integer(state), None)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\npub fn duration_in_tl<'a>(agg: Option<StateAgg<'a>>, state: String) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    duration_in(agg.map(StateAgg::as_compact_state_agg), state)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\npub fn duration_in_tl_int<'a>(agg: Option<StateAgg<'a>>, state: i64) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    duration_in_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        MaterializedState::Integer(state),\n        None,\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_duration_in_string<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorDurationIn,\n) -> crate::raw::Interval {\n    let state = MaterializedState::String(\n        String::from_utf8_lossy(accessor.state_bytes.as_slice()).to_string(),\n    );\n    duration_in_inner(Some(agg.as_compact_state_agg()), state, None)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_duration_in_int<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorDurationInInt,\n) -> crate::raw::Interval {\n    let state = MaterializedState::Integer(accessor.state);\n    duration_in_inner(Some(agg.as_compact_state_agg()), state, None)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\npub fn duration_in_range<'a>(\n    agg: Option<StateAgg<'a>>,\n    state: String,\n    start: TimestampTz,\n    interval: default!(Option<crate::raw::Interval>, \"NULL\"),\n) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    let agg = agg.map(StateAgg::as_compact_state_agg);\n    let interval = interval.map(|interval| crate::datum_utils::interval_to_ms(&start, &interval));\n    let start = start.into();\n    duration_in_inner(\n        agg,\n        MaterializedState::String(state),\n        Some((start, interval)),\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"duration_in\")]\npub fn duration_in_range_int<'a>(\n    agg: Option<StateAgg<'a>>,\n    state: i64,\n    start: TimestampTz,\n    interval: default!(Option<crate::raw::Interval>, \"NULL\"),\n) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    let interval = interval.map(|interval| crate::datum_utils::interval_to_ms(&start, &interval));\n    let start = start.into();\n    duration_in_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        MaterializedState::Integer(state),\n        Some((start, interval)),\n    )\n}\n\n/// Used to indicate no interval was specified. The interval cannot be negative anyways, so this\n/// value will never be a valid argument.\nconst NO_INTERVAL_MARKER: i64 = i64::MIN;\nfn range_tuple(start: i64, interval: i64) -> (i64, Option<i64>) {\n    (\n        start,\n        if interval == NO_INTERVAL_MARKER {\n            None\n        } else {\n            Some(interval)\n        },\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_duration_in_range_string<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorDurationInRange,\n) -> crate::raw::Interval {\n    let state = MaterializedState::String(\n        String::from_utf8_lossy(accessor.state_bytes.as_slice()).to_string(),\n    );\n    duration_in_inner(\n        Some(agg.as_compact_state_agg()),\n        state,\n        Some(range_tuple(accessor.start, accessor.interval)),\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_duration_in_range_int<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorDurationInRangeInt,\n) -> crate::raw::Interval {\n    let state = MaterializedState::Integer(accessor.state);\n    duration_in_inner(\n        Some(agg.as_compact_state_agg()),\n        state,\n        Some(range_tuple(accessor.start, accessor.interval)),\n    )\n}\n\nfn interpolated_duration_in_inner<'a>(\n    aggregate: Option<CompactStateAgg<'a>>,\n    state: MaterializedState,\n    start: i64,\n    interval: i64,\n    prev: Option<CompactStateAgg<'a>>,\n) -> crate::raw::Interval {\n    match aggregate {\n        None => pgrx::error!(\n            \"when interpolating data between grouped data, all groups must contain some data\"\n        ),\n        Some(aggregate) => {\n            if let Some(ref prev) = prev {\n                assert!(\n                    start >= prev.0.last_time,\n                    \"Start time cannot be before last state of previous aggregate\"\n                );\n            };\n            let range = if aggregate.compact {\n                assert!(\n                    start <= aggregate.first_time,\n                    \"For compact state aggregates, the start cannot be after the first state\"\n                );\n                assert!(\n                    (start + interval) >= aggregate.last_time,\n                    \"For compact state aggregates, the time range cannot be after the last state\"\n                );\n                None\n            } else {\n                Some((start, Some(interval)))\n            };\n            let new_agg = aggregate.interpolate(start, interval, prev);\n            duration_in_inner(Some(new_agg), state, range)\n        }\n    }\n}\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn interpolated_duration_in<'a>(\n    agg: Option<CompactStateAgg<'a>>,\n    state: String,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<CompactStateAgg<'a>>,\n) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    interpolated_duration_in_inner(\n        agg,\n        MaterializedState::String(state),\n        start.into(),\n        interval,\n        prev,\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_duration_in\")]\npub fn interpolated_duration_in_tl<'a>(\n    agg: Option<StateAgg<'a>>,\n    state: String,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<StateAgg<'a>>,\n) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    interpolated_duration_in(\n        agg.map(StateAgg::as_compact_state_agg),\n        state,\n        start,\n        interval,\n        prev.map(StateAgg::as_compact_state_agg),\n    )\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    schema = \"toolkit_experimental\",\n    name = \"interpolated_duration_in\"\n)]\npub fn interpolated_duration_in_int<'a>(\n    agg: Option<CompactStateAgg<'a>>,\n    state: i64,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<CompactStateAgg<'a>>,\n) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    interpolated_duration_in_inner(\n        agg,\n        MaterializedState::Integer(state),\n        start.into(),\n        interval,\n        prev,\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_duration_in\")]\npub fn interpolated_duration_in_tl_int<'a>(\n    agg: Option<StateAgg<'a>>,\n    state: i64,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<StateAgg<'a>>,\n) -> crate::raw::Interval {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    interpolated_duration_in_int(\n        agg.map(StateAgg::as_compact_state_agg),\n        state,\n        start,\n        interval,\n        prev.map(StateAgg::as_compact_state_agg),\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_interpolated_duration_in_string<'a>(\n    agg: Option<StateAgg<'a>>,\n    accessor: AccessorInterpolatedDurationIn,\n) -> crate::raw::Interval {\n    let state = MaterializedState::String(\n        String::from_utf8_lossy(accessor.state_bytes.as_slice()).to_string(),\n    );\n    interpolated_duration_in_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        state,\n        accessor.start,\n        accessor.interval,\n        if accessor.prev_present {\n            Some(unsafe { accessor.prev.flatten() }.as_compact_state_agg())\n        } else {\n            None\n        },\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_interpolated_duration_in_int<'a>(\n    agg: Option<StateAgg<'a>>,\n    accessor: AccessorInterpolatedDurationInInt,\n) -> crate::raw::Interval {\n    let state = MaterializedState::Integer(accessor.state);\n    interpolated_duration_in_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        state,\n        accessor.start,\n        accessor.interval,\n        if accessor.prev_present {\n            Some(unsafe { accessor.prev.flatten() }.as_compact_state_agg())\n        } else {\n            None\n        },\n    )\n}\n\nfn duration_in_bad_args_inner() -> ! {\n    panic!(\"The start and interval parameters cannot be used for duration_in with a compact state aggregate\")\n}\n\n#[allow(unused_variables)] // can't underscore-prefix since argument names are used by pgrx\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"duration_in\",\n    schema = \"toolkit_experimental\"\n)]\npub fn duration_in_bad_args<'a>(\n    agg: Option<CompactStateAgg<'a>>,\n    state: String,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n) -> crate::raw::Interval {\n    duration_in_bad_args_inner()\n}\n#[allow(unused_variables)] // can't underscore-prefix since argument names are used by pgrx\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"duration_in\",\n    schema = \"toolkit_experimental\"\n)]\npub fn duration_in_int_bad_args<'a>(\n    agg: Option<CompactStateAgg<'a>>,\n    state: i64,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n) -> crate::raw::Interval {\n    duration_in_bad_args_inner()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn into_values<'a>(\n    agg: CompactStateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(duration, crate::raw::Interval),\n    ),\n> {\n    agg.assert_str();\n    let states: String = agg.states_as_str().to_owned();\n    TableIterator::new(agg.durations.clone().into_iter().map(move |record| {\n        (\n            record.state.as_str(&states).to_string(),\n            record.duration.into(),\n        )\n    }))\n}\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn into_int_values<'a>(\n    agg: CompactStateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(duration, crate::raw::Interval),\n    ),\n> {\n    agg.assert_int();\n    TableIterator::new(\n        agg.durations\n            .clone()\n            .into_iter()\n            .map(move |record| (record.state.into_integer(), record.duration.into()))\n            .collect::<Vec<_>>()\n            .into_iter(), // make map panic now instead of at iteration time\n    )\n}\n#[pg_extern(immutable, parallel_safe, name = \"into_values\")]\npub fn into_values_tl<'a>(\n    agg: StateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(duration, crate::raw::Interval),\n    ),\n> {\n    agg.assert_str();\n    into_values(agg.as_compact_state_agg())\n}\n#[pg_extern(immutable, parallel_safe, name = \"into_int_values\")]\npub fn into_values_tl_int<'a>(\n    agg: StateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(duration, crate::raw::Interval),\n    ),\n> {\n    agg.assert_int();\n    into_int_values(agg.as_compact_state_agg())\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_into_values<'a>(\n    agg: StateAgg<'a>,\n    _accessor: AccessorIntoValues,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(duration, crate::raw::Interval),\n    ),\n> {\n    into_values_tl(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_into_int_values<'a>(\n    agg: StateAgg<'a>,\n    _accessor: AccessorIntoIntValues,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(duration, crate::raw::Interval),\n    ),\n> {\n    into_values_tl_int(agg)\n}\n\nfn state_timeline_inner<'a>(\n    agg: CompactStateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    assert!(\n        !agg.compact,\n        \"state_timeline can only be called on a compact_state_agg built from state_agg\"\n    );\n    let states: String = agg.states_as_str().to_owned();\n    TableIterator::new(\n        agg.combined_durations\n            .clone()\n            .into_iter()\n            .map(move |record| {\n                (\n                    record.state.as_str(&states).to_string(),\n                    TimestampTz::from(record.start_time),\n                    TimestampTz::from(record.end_time),\n                )\n            }),\n    )\n}\nfn state_int_timeline_inner<'a>(\n    agg: CompactStateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    assert!(\n        !agg.compact,\n        \"state_timeline can only be called on a compact_state_agg built from state_agg\"\n    );\n    TableIterator::new(\n        agg.combined_durations\n            .clone()\n            .into_iter()\n            .map(move |record| {\n                (\n                    record.state.into_integer(),\n                    TimestampTz::from(record.start_time),\n                    TimestampTz::from(record.end_time),\n                )\n            })\n            .collect::<Vec<_>>()\n            .into_iter(), // make map panic now instead of at iteration time\n    )\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn state_timeline<'a>(\n    agg: StateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    agg.assert_str();\n    state_timeline_inner(agg.as_compact_state_agg())\n}\n#[pg_extern(immutable, parallel_safe)]\npub fn state_int_timeline<'a>(\n    agg: StateAgg<'a>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    agg.assert_int();\n    state_int_timeline_inner(agg.as_compact_state_agg())\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_state_timeline<'a>(\n    agg: StateAgg<'a>,\n    _accessor: AccessorStateTimeline,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    state_timeline(agg)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_state_int_timeline<'a>(\n    agg: StateAgg<'a>,\n    _accessor: AccessorStateIntTimeline,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    state_int_timeline(agg)\n}\n\nfn interpolated_state_timeline_inner<'a>(\n    agg: Option<StateAgg<'a>>,\n    start: i64,\n    interval: i64,\n    prev: Option<StateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    match agg {\n        None => pgrx::error!(\n            \"when interpolating data between grouped data, all groups must contain some data\"\n        ),\n        Some(agg) => TableIterator::new(\n            state_timeline_inner(agg.as_compact_state_agg().interpolate(\n                start,\n                interval,\n                prev.map(StateAgg::as_compact_state_agg),\n            ))\n            .collect::<Vec<_>>()\n            .into_iter(),\n        ),\n    }\n}\nfn interpolated_state_int_timeline_inner<'a>(\n    agg: Option<StateAgg<'a>>,\n    start: i64,\n    interval: i64,\n    prev: Option<StateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    match agg {\n        None => pgrx::error!(\n            \"when interpolating data between grouped data, all groups must contain some data\"\n        ),\n        Some(agg) => TableIterator::new(\n            state_int_timeline_inner(agg.as_compact_state_agg().interpolate(\n                start,\n                interval,\n                prev.map(StateAgg::as_compact_state_agg),\n            ))\n            .collect::<Vec<_>>()\n            .into_iter(),\n        ),\n    }\n}\n#[pg_extern(immutable, parallel_safe)]\npub fn interpolated_state_timeline<'a>(\n    agg: Option<StateAgg<'a>>,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<StateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    interpolated_state_timeline_inner(agg, start.into(), interval, prev)\n}\n#[pg_extern(immutable, parallel_safe)]\npub fn interpolated_state_int_timeline<'a>(\n    agg: Option<StateAgg<'a>>,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<StateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    interpolated_state_int_timeline_inner(agg, start.into(), interval, prev)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_interpolated_state_timeline<'a>(\n    agg: Option<StateAgg<'a>>,\n    accessor: AccessorInterpolatedStateTimeline,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, String),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    interpolated_state_timeline_inner(\n        agg,\n        accessor.start,\n        accessor.interval,\n        if accessor.prev_present {\n            Some(unsafe { accessor.prev.flatten() })\n        } else {\n            None\n        },\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_interpolated_state_int_timeline<'a>(\n    agg: Option<StateAgg<'a>>,\n    accessor: AccessorInterpolatedStateIntTimeline,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(state, i64),\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    interpolated_state_int_timeline_inner(\n        agg,\n        accessor.start,\n        accessor.interval,\n        if accessor.prev_present {\n            Some(unsafe { accessor.prev.flatten() })\n        } else {\n            None\n        },\n    )\n}\n\nfn state_periods_inner<'a>(\n    agg: CompactStateAgg<'a>,\n    state: MaterializedState,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    assert!(\n        !agg.compact,\n        \"state_periods can only be called on a compact_state_agg built from state_agg\"\n    );\n    let states: String = agg.states_as_str().to_owned();\n    TableIterator::new(\n        agg.combined_durations\n            .clone()\n            .into_iter()\n            .filter_map(move |record| {\n                if record.state.materialize(&states) == state {\n                    Some((\n                        TimestampTz::from(record.start_time),\n                        TimestampTz::from(record.end_time),\n                    ))\n                } else {\n                    None\n                }\n            }),\n    )\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn state_periods<'a>(\n    agg: StateAgg<'a>,\n    state: String,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    agg.assert_str();\n    let agg = agg.as_compact_state_agg();\n    state_periods_inner(agg, MaterializedState::String(state))\n}\n#[pg_extern(immutable, parallel_safe, name = \"state_periods\")]\npub fn state_int_periods<'a>(\n    agg: StateAgg<'a>,\n    state: i64,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    agg.assert_int();\n    state_periods_inner(\n        agg.as_compact_state_agg(),\n        MaterializedState::Integer(state),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_state_periods_string<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorStatePeriods,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    let state = MaterializedState::String(\n        String::from_utf8_lossy(accessor.state_bytes.as_slice()).to_string(),\n    );\n    state_periods_inner(agg.as_compact_state_agg(), state)\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_state_periods_int<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorStatePeriodsInt,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    let state = MaterializedState::Integer(accessor.state);\n    state_periods_inner(agg.as_compact_state_agg(), state)\n}\n\nfn interpolated_state_periods_inner<'a>(\n    aggregate: Option<CompactStateAgg<'a>>,\n    state: MaterializedState,\n    start: i64,\n    interval: i64,\n    prev: Option<CompactStateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    match aggregate {\n        None => pgrx::error!(\n            \"when interpolating data between grouped data, all groups must contain some data\"\n        ),\n        Some(aggregate) => TableIterator::new(\n            state_periods_inner(aggregate.interpolate(start, interval, prev), state)\n                .collect::<Vec<_>>()\n                .into_iter(),\n        ),\n    }\n}\n#[pg_extern(immutable, parallel_safe)]\npub fn interpolated_state_periods<'a>(\n    agg: Option<StateAgg<'a>>,\n    state: String,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<StateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    if let Some(ref agg) = agg {\n        agg.assert_str()\n    };\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    interpolated_state_periods_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        MaterializedState::String(state),\n        start.into(),\n        interval,\n        prev.map(StateAgg::as_compact_state_agg),\n    )\n}\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_state_periods\")]\npub fn interpolated_state_periods_int<'a>(\n    agg: Option<StateAgg<'a>>,\n    state: i64,\n    start: TimestampTz,\n    interval: crate::raw::Interval,\n    prev: Option<StateAgg<'a>>,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    if let Some(ref agg) = agg {\n        agg.assert_int()\n    };\n    let interval = crate::datum_utils::interval_to_ms(&start, &interval);\n    interpolated_state_periods_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        MaterializedState::Integer(state),\n        start.into(),\n        interval,\n        prev.map(StateAgg::as_compact_state_agg),\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_interpolated_state_periods_string<'a>(\n    agg: Option<StateAgg<'a>>,\n    accessor: AccessorInterpolatedStatePeriods,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    let state = MaterializedState::String(\n        String::from_utf8_lossy(accessor.state_bytes.as_slice()).to_string(),\n    );\n    interpolated_state_periods_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        state,\n        accessor.start,\n        accessor.interval,\n        if accessor.prev_present {\n            Some(unsafe { accessor.prev.flatten() }.as_compact_state_agg())\n        } else {\n            None\n        },\n    )\n}\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_interpolated_state_periods_int<'a>(\n    agg: Option<StateAgg<'a>>,\n    accessor: AccessorInterpolatedStatePeriodsInt,\n) -> TableIterator<\n    'a,\n    (\n        pgrx::name!(start_time, TimestampTz),\n        pgrx::name!(end_time, TimestampTz),\n    ),\n> {\n    let state = MaterializedState::Integer(accessor.state);\n    interpolated_state_periods_inner(\n        agg.map(StateAgg::as_compact_state_agg),\n        state,\n        accessor.start,\n        accessor.interval,\n        if accessor.prev_present {\n            Some(unsafe { accessor.prev.flatten() }.as_compact_state_agg())\n        } else {\n            None\n        },\n    )\n}\n\nfn state_at_inner<'a>(agg: StateAgg<'a>, point: i64) -> Option<MaterializedState> {\n    let agg = agg.as_compact_state_agg();\n    let point: i64 = point.into();\n    if agg.combined_durations.is_empty() {\n        return None;\n    }\n\n    // binary search to find the first time at or after the start time\n    let slice = agg.combined_durations.as_slice();\n    let idx = match slice.binary_search_by(|tis| tis.start_time.cmp(&point)) {\n        Ok(idx) => idx,\n        Err(idx) => idx.checked_sub(1)?, // return NULL if before first item\n    };\n    let tis = slice.get(idx).expect(\"binary search index out-of-bounds\");\n\n    Some(tis.state.materialize(agg.states_as_str()))\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"state_at\")]\nfn state_at<'a>(agg: StateAgg<'a>, point: TimestampTz) -> Option<String> {\n    agg.assert_str();\n    state_at_inner(agg, point.into()).map(MaterializedState::into_string)\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"state_at_int\")]\nfn state_at_int<'a>(agg: StateAgg<'a>, point: TimestampTz) -> Option<i64> {\n    agg.assert_int();\n    state_at_inner(agg, point.into()).map(MaterializedState::into_integer)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_state_at_string<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorStateAt,\n) -> Option<String> {\n    state_at_inner(agg, accessor.time).map(MaterializedState::into_string)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_state_agg_state_at_int<'a>(\n    agg: StateAgg<'a>,\n    accessor: AccessorStateAtInt,\n) -> Option<i64> {\n    state_at_inner(agg, accessor.time).map(MaterializedState::into_integer)\n}\n\n#[derive(Clone, Debug, Deserialize, Eq, FlatSerializable, PartialEq, Serialize)]\n#[repr(C)]\npub struct DurationInState {\n    duration: i64,\n    state: StateEntry,\n}\n\n#[derive(Clone, Debug, Deserialize, Eq, FlatSerializable, PartialEq, Serialize)]\n#[repr(C)]\npub struct TimeInState {\n    start_time: i64,\n    end_time: i64,\n    state: StateEntry,\n}\n\nstruct DurationState {\n    last_state: Option<(MaterializedState, i64)>,\n    durations: std::collections::HashMap<MaterializedState, i64>,\n}\nimpl DurationState {\n    fn new() -> Self {\n        Self {\n            last_state: None,\n            durations: std::collections::HashMap::new(),\n        }\n    }\n\n    fn handle_record(&mut self, state: MaterializedState, time: i64) {\n        match self.last_state.take() {\n            None => self.last_state = Some((state, time)),\n            Some((last_state, last_time)) => {\n                debug_assert!(time >= last_time);\n                self.last_state = Some((state, time));\n                match self.durations.get_mut(&last_state) {\n                    None => {\n                        self.durations.insert(last_state, time - last_time);\n                    }\n                    Some(duration) => {\n                        let this_duration = time - last_time;\n                        let new_duration = *duration + this_duration;\n                        *duration = new_duration;\n                    }\n                }\n            }\n        }\n    }\n\n    // It's possible that our last seen state was unique, in which case we'll have to\n    // add a 0 duration entry so that we can handle rollup and interpolation calls\n    fn finalize(&mut self) {\n        if let Some((last_state, _)) = self.last_state.take() {\n            self.durations.entry(last_state).or_insert(0);\n        }\n    }\n}\n\n#[derive(Serialize, Deserialize, Clone, Debug, Eq, Ord, PartialEq, PartialOrd)]\nstruct Record {\n    state: MaterializedState,\n    time: i64,\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use std::sync::atomic::Ordering::Relaxed;\n\n    use super::*;\n    use pgrx_macros::pg_test;\n\n    macro_rules! select_one {\n        ($client:expr, $stmt:expr, $type:ty) => {\n            $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_one::<$type>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    #[pg_test]\n    #[should_panic = \"The start and interval parameters cannot be used for duration_in with\"]\n    fn duration_in_misuse_error() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            assert_eq!(\n                \"365 days 00:02:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one', '2020-01-01', '1 day')::TEXT FROM test\",\n                    &str\n                )\n            );\n        })\n    }\n\n    #[pg_test]\n    fn one_state_one_change() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-12-31 00:02:00+00', 'end')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                \"365 days 00:02:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"365 days 00:02:00\",\n                select_one!(\n                    client,\n                    \"SELECT duration_in(state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn two_states_two_changes() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-01 00:01:00+00', 'two'),\n                    ('2020-12-31 00:02:00+00', 'end')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                \"00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"365 days 00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn two_states_three_changes() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-01 00:01:00+00', 'two'),\n                    ('2020-01-01 00:02:00+00', 'one'),\n                    ('2020-12-31 00:02:00+00', 'end')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                \"365 days 00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n\n            assert_eq!(\n                \"365 days 00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT duration_in(state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT duration_in(state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn out_of_order_times() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-01 00:02:00+00', 'one'),\n                    ('2020-01-01 00:01:00+00', 'two'),\n                    ('2020-12-31 00:02:00+00', 'end')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                \"365 days 00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn same_state_twice() {\n        // TODO Do we care?  Could be that states are recorded not only when they change but\n        // also at regular intervals even when they don't?\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-01 00:01:00+00', 'one'),\n                    ('2020-01-01 00:02:00+00', 'two'),\n                    ('2020-12-31 00:02:00+00', 'end')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                \"00:02:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"365 days\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn duration_in_two_states_two_changes() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-01 00:01:00+00', 'two'),\n                    ('2020-12-31 00:02:00+00', 'end')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                \"00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n            assert_eq!(\n                \"365 days 00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn same_state_twice_last() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-01 00:01:00+00', 'two'),\n                    ('2020-01-01 00:02:00+00', 'two')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                \"00:01:00\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'two')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n    }\n\n    #[pg_test]\n    fn combine_using_muchos_data() {\n        compact_state_agg::counters::reset();\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client.update(\n                r#\"\ninsert into test values ('2020-01-01 00:00:00+00', 'one');\ninsert into test select '2020-01-02 UTC'::timestamptz + make_interval(days=>v), 'two' from generate_series(1,300000) v;\ninsert into test select '2020-01-02 UTC'::timestamptz + make_interval(days=>v), 'three' from generate_series(300001,600000) v;\ninsert into test select '2020-01-02 UTC'::timestamptz + make_interval(days=>v), 'four' from generate_series(600001,900000) v;\n                \"#,\n                None,\n                &[],\n            ).unwrap();\n            assert_eq!(\n                \"2 days\",\n                select_one!(\n                    client,\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one')::TEXT FROM test\",\n                    &str\n                )\n            );\n        });\n        assert!(compact_state_agg::counters::COMBINE_NONE.load(Relaxed) == 0); // TODO untested\n        assert!(compact_state_agg::counters::COMBINE_A.load(Relaxed) == 0); // TODO untested\n        assert!(compact_state_agg::counters::COMBINE_B.load(Relaxed) > 0); // tested\n        assert!(compact_state_agg::counters::COMBINE_BOTH.load(Relaxed) > 0);\n        // tested\n    }\n\n    // TODO This doesn't work under github actions.  Do we run with multiple\n    //   CPUs there?  If not, that would surely make a big difference.\n    // TODO use EXPLAIN to figure out how it differs when run under github actions\n    // #[pg_test]\n    #[allow(dead_code)]\n    fn combine_using_settings() {\n        compact_state_agg::counters::reset();\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'one'),\n                    ('2020-01-03 00:00:00+00', 'two')\n                \"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                \"2 days\",\n                select_one!(\n                    client,\n                    r#\"\nSET parallel_setup_cost = 0;\nSET parallel_tuple_cost = 0;\nSET min_parallel_table_scan_size = 0;\nSET max_parallel_workers_per_gather = 4;\nSET parallel_leader_participation = off;\nSET enable_indexonlyscan = off;\nSELECT toolkit_experimental.duration_in('one', toolkit_experimental.compact_state_agg(ts, state))::TEXT FROM (\n    SELECT * FROM test\n    UNION ALL SELECT * FROM test\n    UNION ALL SELECT * FROM test\n    UNION ALL SELECT * FROM test) u\n                \"#,\n                    &str\n                )\n            );\n        });\n        assert!(compact_state_agg::counters::COMBINE_NONE.load(Relaxed) == 0); // TODO untested\n        assert!(compact_state_agg::counters::COMBINE_A.load(Relaxed) == 0); // TODO untested\n        assert!(compact_state_agg::counters::COMBINE_B.load(Relaxed) > 0); // tested\n        assert!(compact_state_agg::counters::COMBINE_BOTH.load(Relaxed) > 0);\n        // tested\n    }\n\n    // the sample query from the ticket\n    #[pg_test]\n    fn sample_query() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                    ('2020-01-01 00:00:00+00', 'START'),\n                    ('2020-01-01 00:01:00+00', 'ERROR'),\n                    ('2020-01-01 00:02:00+00', 'STOPPED')\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            assert_eq!(\n                client\n                    .update(\n                        r#\"SELECT toolkit_experimental.duration_in(states, 'ERROR')::TEXT as error,\n                                  toolkit_experimental.duration_in(states, 'START')::TEXT as start,\n                                  toolkit_experimental.duration_in(states, 'STOPPED')::TEXT as stopped\n                             FROM (SELECT toolkit_experimental.compact_state_agg(ts, state) as states FROM test) as foo\"#,\n                        None,\n                        &[],\n                    )\n                    .unwrap().first()\n                    .get_three::<&str, &str, &str>().unwrap(),\n                (Some(\"00:01:00\"), Some(\"00:01:00\"), Some(\"00:00:00\"))\n            );\n            assert_eq!(\n                client\n                    .update(\n                        r#\"SELECT duration_in(states, 'ERROR')::TEXT as error,\n                                  duration_in(states, 'START')::TEXT as start,\n                                  duration_in(states, 'STOPPED')::TEXT as stopped\n                             FROM (SELECT state_agg(ts, state) as states FROM test) as foo\"#,\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_three::<&str, &str, &str>()\n                    .unwrap(),\n                (Some(\"00:01:00\"), Some(\"00:01:00\"), Some(\"00:00:00\"))\n            );\n        })\n    }\n\n    #[pg_test]\n    fn interpolated_duration() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"SET TIME ZONE 'UTC';\n                CREATE TABLE inttest(time TIMESTAMPTZ, state TEXT, bucket INT);\n                CREATE TABLE inttest2(time TIMESTAMPTZ, state BIGINT, bucket INT);\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO inttest VALUES\n                ('2020-1-1 10:00'::timestamptz, 'one', 1),\n                ('2020-1-1 12:00'::timestamptz, 'two', 1), \n                ('2020-1-1 16:00'::timestamptz, 'three', 1), \n                ('2020-1-2 2:00'::timestamptz, 'one', 2), \n                ('2020-1-2 12:00'::timestamptz, 'two', 2), \n                ('2020-1-2 20:00'::timestamptz, 'three', 2), \n                ('2020-1-3 10:00'::timestamptz, 'one', 3), \n                ('2020-1-3 12:00'::timestamptz, 'two', 3), \n                ('2020-1-3 16:00'::timestamptz, 'three', 3);\n                INSERT INTO inttest2 VALUES\n                ('2020-1-1 10:00'::timestamptz, 10001, 1),\n                ('2020-1-1 12:00'::timestamptz, 10002, 1), \n                ('2020-1-1 16:00'::timestamptz, 10003, 1), \n                ('2020-1-2 2:00'::timestamptz, 10001, 2), \n                ('2020-1-2 12:00'::timestamptz, 10002, 2), \n                ('2020-1-2 20:00'::timestamptz, 10003, 2), \n                ('2020-1-3 10:00'::timestamptz, 10001, 3), \n                ('2020-1-3 12:00'::timestamptz, 10002, 3), \n                ('2020-1-3 16:00'::timestamptz, 10003, 3);\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Interpolate time spent in state \"three\" each day\n            let mut durations = client.update(\n                r#\"SELECT\n                toolkit_experimental.interpolated_duration_in(\n                    agg, \n                    'three',\n                    '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket)\n                )::TEXT FROM (\n                    SELECT bucket, toolkit_experimental.compact_state_agg(time, state) as agg \n                    FROM inttest \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                None,\n                &[],\n            ).unwrap();\n\n            // Day 1, in \"three\" from \"16:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"08:00:00\")\n            );\n            // Day 2, in \"three\" from start of day to \"2:00\" and \"20:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"06:00:00\")\n            );\n            // Day 3, in \"three\" from start of day to end\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"18:00:00\")\n            );\n            assert!(durations.next().is_none());\n\n            let mut durations = client.update(\n                r#\"SELECT\n                interpolated_duration_in(\n                    agg,\n                    'three', \n                    '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket)\n                )::TEXT FROM (\n                    SELECT bucket, state_agg(time, state) as agg \n                    FROM inttest \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                None,\n                &[],\n            ).unwrap();\n\n            // Day 1, in \"three\" from \"16:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"08:00:00\")\n            );\n            // Day 2, in \"three\" from start of day to \"2:00\" and \"20:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"06:00:00\")\n            );\n            // Day 3, in \"three\" from start of day to end\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"18:00:00\")\n            );\n            assert!(durations.next().is_none());\n\n            let mut durations = client.update(\n                r#\"SELECT\n                interpolated_duration_in(\n                    agg,\n                    10003, \n                    '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket)\n                )::TEXT FROM (\n                    SELECT bucket, state_agg(time, state) as agg \n                    FROM inttest2 \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                None,\n                &[],\n            ).unwrap();\n\n            // Day 1, in \"three\" from \"16:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"08:00:00\")\n            );\n            // Day 2, in \"three\" from start of day to \"2:00\" and \"20:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"06:00:00\")\n            );\n            // Day 3, in \"three\" from start of day to end\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"18:00:00\")\n            );\n            assert!(durations.next().is_none());\n\n            let mut durations = client.update(\n                r#\"SELECT\n                toolkit_experimental.interpolated_duration_in(\n                    agg,\n                    10003,\n                    '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket)\n                )::TEXT FROM (\n                    SELECT bucket, toolkit_experimental.compact_state_agg(time, state) as agg \n                    FROM inttest2\n                    GROUP BY bucket ORDER BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                None,\n                &[],\n            ).unwrap();\n\n            // Day 1, in \"three\" from \"16:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"08:00:00\")\n            );\n            // Day 2, in \"three\" from start of day to \"2:00\" and \"20:00\" to end of day\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"06:00:00\")\n            );\n            // Day 3, in \"three\" from start of day to end\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"18:00:00\")\n            );\n            assert!(durations.next().is_none());\n        });\n    }\n\n    #[pg_test(\n        error = \"state cannot be both String(\\\"ERROR\\\") and String(\\\"START\\\") at 631152000000000\"\n    )]\n    fn two_states_at_one_time() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test(ts timestamptz, state TEXT)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                        ('2020-01-01 00:00:00+00', 'START'),\n                        ('2020-01-01 00:00:00+00', 'ERROR')\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"SELECT toolkit_experimental.duration_in(toolkit_experimental.compact_state_agg(ts, state), 'one') FROM test\",\n                    None,\n                    &[]\n                )\n                .unwrap();\n            client\n                .update(\n                    \"SELECT duration_in(state_agg(ts, state), 'one') FROM test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n        })\n    }\n\n    #[pg_test]\n    fn interpolate_introduces_state() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE states(time TIMESTAMPTZ, state TEXT, bucket INT)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO states VALUES\n                ('2020-1-1 10:00', 'starting', 1),\n                ('2020-1-1 10:30', 'running', 1),\n                ('2020-1-2 16:00', 'error', 2),\n                ('2020-1-3 18:30', 'starting', 3),\n                ('2020-1-3 19:30', 'running', 3),\n                ('2020-1-4 12:00', 'stopping', 4)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut durations = client\n                .update(\n                    r#\"SELECT \n                toolkit_experimental.interpolated_duration_in(\n                    agg,\n                    'running',\n                  '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval,\n                  LAG(agg) OVER (ORDER BY bucket)\n                )::TEXT FROM (\n                    SELECT bucket, toolkit_experimental.compact_state_agg(time, state) as agg\n                    FROM states\n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"13:30:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"16:00:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"04:30:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"12:00:00\")\n            );\n\n            let mut durations = client\n                .update(\n                    r#\"SELECT \n                interpolated_duration_in(\n                    agg,\n                    'running',\n                  '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval,\n                  LAG(agg) OVER (ORDER BY bucket)\n                )::TEXT FROM (\n                    SELECT bucket, state_agg(time, state) as agg\n                    FROM states\n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"13:30:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"16:00:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"04:30:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"12:00:00\")\n            );\n\n            let mut durations = client\n                .update(\n                    r#\"SELECT \n                (agg -> interpolated_duration_in(\n                    'running',\n                    '2019-12-31 0:00'::timestamptz + (bucket * '1 day'::interval), '1 day'::interval,\n                    LAG(agg) OVER (ORDER BY bucket)\n                ))::TEXT FROM (\n                    SELECT bucket, state_agg(time, state) as agg\n                    FROM states\n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"13:30:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"16:00:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"04:30:00\")\n            );\n            assert_eq!(\n                durations.next().unwrap()[1].value().unwrap(),\n                Some(\"12:00:00\")\n            );\n        })\n    }\n\n    #[pg_test]\n    fn text_serialization() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"SET TIME ZONE 'UTC';\n                    CREATE TABLE states(ts TIMESTAMPTZ, state TEXT);\n                    CREATE TABLE states_int(ts TIMESTAMPTZ, state BIGINT);\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            // only a single entry so ordering is consistent between runs\n            client\n                .update(\n                    r#\"INSERT INTO states VALUES\n                        ('2020-1-1 10:00', 'starting');\n                    INSERT INTO states_int VALUES\n                        ('2020-1-1 10:00', -67876545);\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let agg_text = select_one!(\n                client,\n                \"SELECT state_agg(ts, state)::TEXT FROM states\",\n                &str\n            );\n            let expected = \"(version:1,compact_state_agg:(version:1,states_len:8,durations_len:1,durations:[(duration:0,state:(a:0,b:8))],combined_durations_len:1,combined_durations:[(start_time:631188000000000,end_time:631188000000000,state:(a:0,b:8))],first_time:631188000000000,last_time:631188000000000,first_state:0,last_state:0,states:[115,116,97,114,116,105,110,103],compact:false,integer_states:false))\";\n            assert_eq!(agg_text, expected);\n\n            let agg_text = select_one!(\n                client,\n                \"SELECT state_agg(ts, state)::TEXT FROM states_int\",\n                &str\n            );\n            let expected = \"(version:1,compact_state_agg:(version:1,states_len:0,durations_len:1,durations:[(duration:0,state:(a:9223372036854775807,b:-67876545))],combined_durations_len:1,combined_durations:[(start_time:631188000000000,end_time:631188000000000,state:(a:9223372036854775807,b:-67876545))],first_time:631188000000000,last_time:631188000000000,first_state:0,last_state:0,states:[],compact:false,integer_states:true))\";\n            assert_eq!(agg_text, expected);\n        });\n    }\n\n    #[pg_test]\n    fn combine() {\n        assert_eq!(state_agg::combine(None, None), None);\n\n        let mut trans_state_2 = CompactStateAggTransState::new(true);\n        trans_state_2.record(MaterializedState::Integer(444), 10005000);\n        let mut trans_state_1 = CompactStateAggTransState::new(true);\n        trans_state_1.record(MaterializedState::Integer(333), 10000000);\n        let trans_state = state_agg::combine(Some(&trans_state_1), Some(&trans_state_2)).unwrap();\n        let trans_state = state_agg::combine(Some(&trans_state), None).unwrap();\n        let trans_state = state_agg::combine(None, Some(&trans_state)).unwrap();\n        assert_eq!(\n            trans_state,\n            CompactStateAggTransState {\n                records: vec![\n                    Record {\n                        state: MaterializedState::Integer(333),\n                        time: 10000000\n                    },\n                    Record {\n                        state: MaterializedState::Integer(444),\n                        time: 10005000\n                    }\n                ],\n                integer_states: true,\n            }\n        );\n    }\n\n    #[pg_test]\n    fn binary_serialization_integer() {\n        let mut trans_state = CompactStateAggTransState::new(true);\n        // only inserting one state since to avoid random ordering\n        trans_state.record(MaterializedState::Integer(22), 99);\n        let agg = state_agg::finally(Some(&mut trans_state)).unwrap();\n\n        // dis: duration i64, state entry (i64, i64)\n        let expected = [\n            232, 1, 0, 0, // header\n            1, // version\n            0, 0, 0, // padding\n            // inner compact_state_agg:\n            200, 1, 0, 0, // header\n            1, // version\n            0, 0, 0, // padding\n            0, 0, 0, 0, 0, 0, 0, 0, // states_len (empty since integer states)\n            1, 0, 0, 0, 0, 0, 0, 0, // durations_len\n            0, 0, 0, 0, 0, 0, 0, 0, // state 1: duration\n            255, 255, 255, 255, 255, 255, 255, 127, // state 1:  a\n            22, 0, 0, 0, 0, 0, 0, 0, // state 1: b\n            1, 0, 0, 0, 0, 0, 0, 0, // combined_durations_len\n            99, 0, 0, 0, 0, 0, 0, 0, // state 1: start time\n            99, 0, 0, 0, 0, 0, 0, 0, // state 1: end time\n            255, 255, 255, 255, 255, 255, 255, 127, // state 1: a\n            22, 0, 0, 0, 0, 0, 0, 0, // state 1: b\n            99, 0, 0, 0, 0, 0, 0, 0, // first_time\n            99, 0, 0, 0, 0, 0, 0, 0, // last_time\n            0, 0, 0, 0, // first_state (index)\n            0, 0, 0, 0, // last_state (index)\n            // states array is empty\n            0, // compact (false)\n            1, // integer states (true)\n        ];\n        assert_eq!(agg.to_pg_bytes(), expected);\n    }\n\n    #[pg_test]\n    fn binary_serialization_string() {\n        let mut trans_state = CompactStateAggTransState::new(false);\n        // only inserting one state since to avoid random ordering\n        trans_state.record(MaterializedState::String(\"ABC\".to_string()), 99);\n        let agg = state_agg::finally(Some(&mut trans_state)).unwrap();\n\n        // dis: duration i64, state entry (i64, i64)\n        let expected = [\n            244, 1, 0, 0, // header\n            1, // version\n            0, 0, 0, // padding\n            // inner compact_state_agg:\n            212, 1, 0, 0, // header\n            1, // version\n            0, 0, 0, // padding\n            3, 0, 0, 0, 0, 0, 0, 0, // states_len\n            1, 0, 0, 0, 0, 0, 0, 0, // durations_len\n            0, 0, 0, 0, 0, 0, 0, 0, // state 1: duration\n            0, 0, 0, 0, 0, 0, 0, 0, // state 1: a\n            3, 0, 0, 0, 0, 0, 0, 0, // state 1: b\n            1, 0, 0, 0, 0, 0, 0, 0, // combined_durations_len\n            99, 0, 0, 0, 0, 0, 0, 0, // state 1: start time\n            99, 0, 0, 0, 0, 0, 0, 0, // state 1: end time\n            0, 0, 0, 0, 0, 0, 0, 0, // state 1: a\n            3, 0, 0, 0, 0, 0, 0, 0, // state 1: b\n            99, 0, 0, 0, 0, 0, 0, 0, // first_time\n            99, 0, 0, 0, 0, 0, 0, 0, // last_time\n            0, 0, 0, 0, // first_state (index)\n            0, 0, 0, 0, // last_state (index)\n            65, 66, 67, // states array\n            0,  // compact (false)\n            0,  // integer states (false)\n        ];\n        assert_eq!(agg.to_pg_bytes(), expected);\n    }\n}\n"
  },
  {
    "path": "extension/src/stats_agg.rs",
    "content": "use pgrx::*;\n\nuse crate::{\n    accessors::{\n        AccessorAverage, AccessorAverageX, AccessorAverageY, AccessorCorr, AccessorCovar,\n        AccessorDeterminationCoeff, AccessorIntercept, AccessorKurtosis, AccessorKurtosisX,\n        AccessorKurtosisY, AccessorNumVals, AccessorSkewness, AccessorSkewnessX, AccessorSkewnessY,\n        AccessorSlope, AccessorStdDev, AccessorStdDevX, AccessorStdDevY, AccessorSum, AccessorSumX,\n        AccessorSumY, AccessorVariance, AccessorVarianceX, AccessorVarianceY, AccessorXIntercept,\n    },\n    aggregate_utils::in_aggregate_context,\n    build,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\npub use stats_agg::stats1d::StatsSummary1D as InternalStatsSummary1D;\npub use stats_agg::stats2d::StatsSummary2D as InternalStatsSummary2D;\nuse stats_agg::XYPair;\n\nuse crate::stats_agg::Method::*;\nuse stats_agg::TwoFloat;\n\nuse crate::raw::bytea;\n\ntype StatsSummary1DTF = InternalStatsSummary1D<TwoFloat>;\ntype StatsSummary2DTF = InternalStatsSummary2D<TwoFloat>;\n\npg_type! {\n    #[derive(Debug, PartialEq)]\n    struct StatsSummary1D {\n        n: u64,\n        sx: f64,\n        sx2: f64,\n        sx3: f64,\n        sx4: f64,\n    }\n}\n\npg_type! {\n    #[derive(Debug, PartialEq)]\n    struct StatsSummary2D {\n        n: u64,\n        sx: f64,\n        sx2: f64,\n        sx3: f64,\n        sx4: f64,\n        sy: f64,\n        sy2: f64,\n        sy3: f64,\n        sy4: f64,\n        sxy: f64,\n    }\n}\n\nron_inout_funcs!(StatsSummary1D);\nron_inout_funcs!(StatsSummary2D);\n\nimpl StatsSummary1D {\n    fn to_internal(&self) -> InternalStatsSummary1D<f64> {\n        InternalStatsSummary1D {\n            n: self.n,\n            sx: self.sx,\n            sx2: self.sx2,\n            sx3: self.sx3,\n            sx4: self.sx4,\n        }\n    }\n    pub fn from_internal(st: InternalStatsSummary1D<f64>) -> Self {\n        build!(StatsSummary1D {\n            n: st.n,\n            sx: st.sx,\n            sx2: st.sx2,\n            sx3: st.sx3,\n            sx4: st.sx4,\n        })\n    }\n}\n\nimpl StatsSummary2D {\n    fn to_internal(&self) -> InternalStatsSummary2D<f64> {\n        InternalStatsSummary2D {\n            n: self.n,\n            sx: self.sx,\n            sx2: self.sx2,\n            sx3: self.sx3,\n            sx4: self.sx4,\n            sy: self.sy,\n            sy2: self.sy2,\n            sy3: self.sy3,\n            sy4: self.sy4,\n            sxy: self.sxy,\n        }\n    }\n    fn from_internal(st: InternalStatsSummary2D<f64>) -> Self {\n        build!(StatsSummary2D {\n            n: st.n,\n            sx: st.sx,\n            sx2: st.sx2,\n            sx3: st.sx3,\n            sx4: st.sx4,\n            sy: st.sy,\n            sy2: st.sy2,\n            sy3: st.sy3,\n            sy4: st.sy4,\n            sxy: st.sxy,\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn stats1d_trans_serialize(state: Internal) -> bytea {\n    let ser: &StatsSummary1D = unsafe { state.get().unwrap() };\n    let ser: &StatsSummary1DData = &ser.0;\n    crate::do_serialize!(ser)\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn stats1d_trans_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    stats1d_trans_deserialize_inner(bytes).internal()\n}\npub fn stats1d_trans_deserialize_inner(bytes: bytea) -> Inner<StatsSummary1D> {\n    let de: StatsSummary1D = crate::do_deserialize!(bytes, StatsSummary1DData);\n    de.into()\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn stats2d_trans_serialize(state: Internal) -> bytea {\n    let ser: &StatsSummary2D = unsafe { state.get().unwrap() };\n    let ser: &StatsSummary2DData = &ser.0;\n    crate::do_serialize!(ser)\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn stats2d_trans_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    stats2d_trans_deserialize_inner(bytes).internal()\n}\npub fn stats2d_trans_deserialize_inner(bytes: bytea) -> Inner<StatsSummary2D> {\n    let de: StatsSummary2D = crate::do_deserialize!(bytes, StatsSummary2DData);\n    de.into()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats1d_trans(\n    state: Internal,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats1d_trans_inner(unsafe { state.to_inner() }, val, fcinfo).internal()\n}\n#[pg_extern(immutable, parallel_safe)]\npub fn stats1d_tf_trans(\n    state: Internal,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats1d_tf_trans_inner(unsafe { state.to_inner() }, val, fcinfo).internal()\n}\npub fn stats1d_trans_inner(\n    state: Option<Inner<StatsSummary1D>>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state, val) {\n                (None, None) => {\n                    Some(StatsSummary1D::from_internal(InternalStatsSummary1D::new()).into())\n                } // return an empty one from the trans function because otherwise it breaks in the window context\n                (Some(state), None) => Some(state),\n                (None, Some(val)) => {\n                    let mut s = InternalStatsSummary1D::new();\n                    s.accum(val).unwrap();\n                    Some(StatsSummary1D::from_internal(s).into())\n                }\n                (Some(mut state), Some(val)) => {\n                    let mut s: InternalStatsSummary1D<f64> = state.to_internal();\n                    s.accum(val).unwrap();\n                    *state = StatsSummary1D::from_internal(s);\n                    Some(state)\n                }\n            }\n        })\n    }\n}\npub fn stats1d_tf_trans_inner(\n    state: Option<Inner<StatsSummary1DTF>>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1DTF>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state, val) {\n                (None, None) => Some(InternalStatsSummary1D::new().into()), // return an empty one from the trans function because otherwise it breaks in the window context\n                (Some(state), None) => Some(state),\n                (None, Some(val)) => {\n                    let val = TwoFloat::from(val);\n                    let mut s = InternalStatsSummary1D::new();\n                    s.accum(val).unwrap();\n                    Some(s.into())\n                }\n                (Some(mut state), Some(val)) => {\n                    let val = TwoFloat::from(val);\n                    state.accum(val).unwrap();\n                    Some(state)\n                }\n            }\n        })\n    }\n}\n\n// Note that in general, for all stats2d cases, if either the y or x value is missing, we disregard the entire point as the n is shared between them\n// if the user wants us to treat nulls as a particular value (ie zero), they can use COALESCE to do so\n#[pg_extern(immutable, parallel_safe)]\npub fn stats2d_trans(\n    state: Internal,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats2d_trans_inner(unsafe { state.to_inner() }, y, x, fcinfo).internal()\n}\npub fn stats2d_trans_inner(\n    state: Option<Inner<StatsSummary2D>>,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let val: Option<XYPair<f64>> = match (y, x) {\n                (None, _) => None,\n                (_, None) => None,\n                (Some(y), Some(x)) => Some(XYPair { y, x }),\n            };\n            match (state, val) {\n                (None, None) => {\n                    // return an empty one from the trans function because otherwise it breaks in the window context\n                    Some(StatsSummary2D::from_internal(InternalStatsSummary2D::new()).into())\n                }\n                (Some(state), None) => Some(state),\n                (None, Some(val)) => {\n                    let mut s = InternalStatsSummary2D::new();\n                    s.accum(val).unwrap();\n                    Some(StatsSummary2D::from_internal(s).into())\n                }\n                (Some(mut state), Some(val)) => {\n                    let mut s: InternalStatsSummary2D<f64> = state.to_internal();\n                    s.accum(val).unwrap();\n                    *state = StatsSummary2D::from_internal(s);\n                    Some(state)\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats2d_tf_trans(\n    state: Internal,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats2d_tf_trans_inner(unsafe { state.to_inner() }, y, x, fcinfo).internal()\n}\npub fn stats2d_tf_trans_inner(\n    state: Option<Inner<StatsSummary2DTF>>,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2DTF>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let val: Option<XYPair<TwoFloat>> = match (y, x) {\n                (None, _) => None,\n                (_, None) => None,\n                (Some(y), Some(x)) => Some(XYPair {\n                    y: y.into(),\n                    x: x.into(),\n                }),\n            };\n            match (state, val) {\n                (None, None) => {\n                    // return an empty one from the trans function because otherwise it breaks in the window context\n                    Some(StatsSummary2DTF::new().into())\n                }\n                (Some(state), None) => Some(state),\n                (None, Some(val)) => {\n                    let mut s = InternalStatsSummary2D::new();\n                    s.accum(val).unwrap();\n                    Some(s.into())\n                }\n                (Some(mut state), Some(val)) => {\n                    let mut s: StatsSummary2DTF = *state;\n                    s.accum(val).unwrap();\n                    *state = s;\n                    Some(state)\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable)]\npub fn stats1d_inv_trans(\n    state: Internal,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats1d_inv_trans_inner(unsafe { state.to_inner() }, val, fcinfo).internal()\n}\npub fn stats1d_inv_trans_inner(\n    state: Option<Inner<StatsSummary1D>>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, val) {\n            (None, _) => panic!(\"Inverse function should never be called with NULL state\"),\n            (Some(state), None) => Some(state),\n            (Some(state), Some(val)) => {\n                let s: InternalStatsSummary1D<f64> = state.to_internal();\n                let s = s.remove(val);\n                s.map(|s| StatsSummary1D::from_internal(s).into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable)]\npub fn stats1d_tf_inv_trans(\n    state: Internal,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats1d_tf_inv_trans_inner(unsafe { state.to_inner() }, val, fcinfo).internal()\n}\npub fn stats1d_tf_inv_trans_inner(\n    state: Option<Inner<StatsSummary1DTF>>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1DTF>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, val) {\n            (None, _) => panic!(\"Inverse function should never be called with NULL state\"),\n            (Some(state), None) => Some(state),\n            (Some(state), Some(val)) => {\n                let val = TwoFloat::new_add(val, 0.0);\n                let state = state.remove(val);\n                state.map(|s| s.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable)]\npub fn stats2d_inv_trans(\n    state: Internal,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats2d_inv_trans_inner(unsafe { state.to_inner() }, y, x, fcinfo).internal()\n}\npub fn stats2d_inv_trans_inner(\n    state: Option<Inner<StatsSummary2D>>,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let val: Option<XYPair<f64>> = match (y, x) {\n                (None, _) => None,\n                (_, None) => None,\n                (Some(y), Some(x)) => Some(XYPair { y, x }),\n            };\n            match (state, val) {\n                (None, _) => panic!(\"Inverse function should never be called with NULL state\"),\n                (Some(state), None) => Some(state),\n                (Some(state), Some(val)) => {\n                    let s: InternalStatsSummary2D<f64> = state.to_internal();\n                    let s = s.remove(val);\n                    s.map(|s| StatsSummary2D::from_internal(s).into())\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable)]\npub fn stats2d_tf_inv_trans(\n    state: Internal,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats2d_tf_inv_trans_inner(unsafe { state.to_inner() }, y, x, fcinfo).internal()\n}\npub fn stats2d_tf_inv_trans_inner(\n    state: Option<Inner<StatsSummary2DTF>>,\n    y: Option<f64>,\n    x: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2DTF>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let val: Option<XYPair<TwoFloat>> = match (y, x) {\n                (None, _) => None,\n                (_, None) => None,\n                (Some(y), Some(x)) => Some(XYPair {\n                    y: y.into(),\n                    x: x.into(),\n                }),\n            };\n            match (state, val) {\n                (None, _) => panic!(\"Inverse function should never be called with NULL state\"),\n                (Some(state), None) => Some(state),\n                (Some(state), Some(val)) => {\n                    let s: InternalStatsSummary2D<TwoFloat> = *state;\n                    let s = s.remove(val);\n                    s.map(|s| s.into())\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats1d_summary_trans(\n    state: Internal,\n    value: Option<StatsSummary1D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats1d_summary_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn stats1d_summary_trans_inner(\n    state: Option<Inner<StatsSummary1D>>,\n    value: Option<StatsSummary1D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, value) {\n            (state, None) => state,\n            (None, Some(value)) => Some(value.in_current_context().into()),\n            (Some(state), Some(value)) => {\n                let s = state.to_internal();\n                let v = value.to_internal();\n                let s = s.combine(v).unwrap();\n                let s = StatsSummary1D::from_internal(s);\n                Some(s.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats2d_summary_trans(\n    state: Internal,\n    value: Option<StatsSummary2D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats2d_summary_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn stats2d_summary_trans_inner(\n    state: Option<Inner<StatsSummary2D>>,\n    value: Option<StatsSummary2D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, value) {\n            (state, None) => state,\n            (None, Some(value)) => Some(value.in_current_context().into()),\n            (Some(state), Some(value)) => {\n                let s = state.to_internal();\n                let v = value.to_internal();\n                let s = s.combine(v).unwrap();\n                let s = StatsSummary2D::from_internal(s);\n                Some(s.into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats1d_summary_inv_trans(\n    state: Internal,\n    value: Option<StatsSummary1D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats1d_summary_inv_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn stats1d_summary_inv_trans_inner(\n    state: Option<Inner<StatsSummary1D>>,\n    value: Option<StatsSummary1D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, &value) {\n            (None, _) => panic!(\"Inverse function should never be called with NULL state\"),\n            (Some(state), None) => Some(state),\n            (Some(state), Some(value)) => {\n                let s = state.to_internal();\n                let v = value.to_internal();\n                let s = s.remove_combined(v);\n                s.map(|s| StatsSummary1D::from_internal(s).into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats2d_summary_inv_trans(\n    state: Internal,\n    value: Option<StatsSummary2D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    stats2d_summary_inv_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn stats2d_summary_inv_trans_inner(\n    state: Option<Inner<StatsSummary2D>>,\n    value: Option<StatsSummary2D>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, &value) {\n            (None, _) => panic!(\"Inverse function should never be called with NULL state\"),\n            (Some(state), None) => Some(state),\n            (Some(state), Some(value)) => {\n                let s = state.to_internal();\n                let v = value.to_internal();\n                let s = s.remove_combined(v);\n                s.map(|s| StatsSummary2D::from_internal(s).into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats1d_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { stats1d_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\npub fn stats1d_combine_inner(\n    state1: Option<Inner<StatsSummary1D>>,\n    state2: Option<Inner<StatsSummary1D>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary1D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(state2)) => {\n                let s = state2.in_current_context();\n                Some(s.into())\n            }\n            (Some(state1), None) => {\n                let s = state1.in_current_context();\n                Some(s.into())\n            }\n            (Some(state1), Some(state2)) => {\n                let s1 = state1.to_internal();\n                let s2 = state2.to_internal();\n                let s1 = s1.combine(s2).unwrap();\n                Some(StatsSummary1D::from_internal(s1).into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn stats2d_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { stats2d_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\npub fn stats2d_combine_inner(\n    state1: Option<Inner<StatsSummary2D>>,\n    state2: Option<Inner<StatsSummary2D>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<StatsSummary2D>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(state2)) => {\n                let s = state2.in_current_context();\n                Some(s.into())\n            }\n            (Some(state1), None) => {\n                let s = state1.in_current_context();\n                Some(s.into())\n            }\n            (Some(state1), Some(state2)) => {\n                let s1 = state1.to_internal();\n                let s2 = state2.to_internal();\n                let s1 = s1.combine(s2).unwrap();\n                Some(StatsSummary2D::from_internal(s1).into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn stats1d_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<StatsSummary1D> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match state.get() {\n            None => None,\n            Some(state) => {\n                let state: &StatsSummary1D = state;\n                Some(state.in_current_context())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn stats1d_tf_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n    // return a normal stats summary here\n) -> Option<StatsSummary1D> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match state.get() {\n            None => None,\n            Some(state) => {\n                let state: &StatsSummary1DTF = state;\n                let state: InternalStatsSummary1D<TwoFloat> = *state;\n                let state: InternalStatsSummary1D<f64> = state.into();\n                let state: StatsSummary1D = StatsSummary1D::from_internal(state);\n                Some(state.in_current_context())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn stats2d_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<StatsSummary2D> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match state.get() {\n            None => None,\n            Some(state) => {\n                let state: &StatsSummary2D = state;\n                Some(state.in_current_context())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn stats2d_tf_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<StatsSummary2D> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match state.get() {\n            None => None,\n            Some(state) => {\n                let state: StatsSummary2DTF = *state;\n                let state: InternalStatsSummary2D<f64> = state.into();\n                let state: StatsSummary2D = StatsSummary2D::from_internal(state);\n                Some(state.in_current_context())\n            }\n        })\n    }\n}\n\n// no serial/unserial/combine function for TwoFloats since moving aggregate mode and partial aggregate mode are mutually exclusiveca f\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE stats_agg( value DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = stats1d_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats1d_final,\\n\\\n        combinefunc = stats1d_combine,\\n\\\n        serialfunc = stats1d_trans_serialize,\\n\\\n        deserialfunc = stats1d_trans_deserialize,\\n\\\n        msfunc = stats1d_tf_trans,\\n\\\n        minvfunc = stats1d_tf_inv_trans,\\n\\\n        mstype = internal,\\n\\\n        mfinalfunc = stats1d_tf_final,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_agg_1d\",\n    requires = [\n        stats1d_trans,\n        stats1d_final,\n        stats1d_combine,\n        stats1d_trans_serialize,\n        stats1d_trans_deserialize,\n        stats1d_trans,\n        stats1d_inv_trans,\n        stats1d_final\n    ],\n);\n\nextension_sql!(\n    \"CREATE AGGREGATE toolkit_experimental.stats_agg_tf( value DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = stats1d_tf_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats1d_tf_final,\\n\\\n        msfunc = stats1d_tf_trans,\\n\\\n        minvfunc = stats1d_tf_inv_trans,\\n\\\n        mstype = internal,\\n\\\n        mfinalfunc = stats1d_tf_final,\\n\\\n        parallel = safe\\n\\\n    );\",\n    name = \"stats_agg_tf_1d\",\n    requires = [\n        stats1d_tf_trans,\n        stats1d_tf_final,\n        stats1d_tf_trans,\n        stats1d_tf_inv_trans,\n        stats1d_tf_final\n    ],\n);\n\n// mostly for testing/debugging, in case we want one without the inverse functions defined.\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE stats_agg_no_inv( value DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = stats1d_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats1d_final,\\n\\\n        combinefunc = stats1d_combine,\\n\\\n        serialfunc = stats1d_trans_serialize,\\n\\\n        deserialfunc = stats1d_trans_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_agg_no_inv\",\n    requires = [\n        stats1d_trans,\n        stats1d_final,\n        stats1d_combine,\n        stats1d_trans_serialize,\n        stats1d_trans_deserialize\n    ],\n);\n\n// same things for the 2d case\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE stats_agg( y DOUBLE PRECISION, x DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = stats2d_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats2d_final,\\n\\\n        combinefunc = stats2d_combine,\\n\\\n        serialfunc = stats2d_trans_serialize,\\n\\\n        deserialfunc = stats2d_trans_deserialize,\\n\\\n        msfunc = stats2d_tf_trans,\\n\\\n        minvfunc = stats2d_tf_inv_trans,\\n\\\n        mstype = internal,\\n\\\n        mfinalfunc = stats2d_tf_final,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_agg_2d\",\n    requires = [\n        stats2d_trans,\n        stats2d_final,\n        stats2d_combine,\n        stats2d_trans_serialize,\n        stats2d_trans_deserialize,\n        stats2d_tf_trans,\n        stats2d_tf_inv_trans,\n        stats2d_tf_final,\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE toolkit_experimental.stats_agg_tf( y DOUBLE PRECISION, x DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = stats2d_tf_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats2d_tf_final,\\n\\\n        msfunc = stats2d_tf_trans,\\n\\\n        minvfunc = stats2d_tf_inv_trans,\\n\\\n        mstype = internal,\\n\\\n        mfinalfunc = stats2d_tf_final,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_agg_2d_tf\",\n    requires = [stats2d_tf_trans, stats2d_tf_inv_trans, stats2d_tf_final],\n);\n\n// mostly for testing/debugging, in case we want one without the inverse functions defined.\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE stats_agg_no_inv( y DOUBLE PRECISION, x DOUBLE PRECISION )\\n\\\n    (\\n\\\n        sfunc = stats2d_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats2d_final,\\n\\\n        combinefunc = stats2d_combine,\\n\\\n        serialfunc = stats2d_trans_serialize,\\n\\\n        deserialfunc = stats2d_trans_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_agg2_no_inv\",\n    requires = [\n        stats2d_trans,\n        stats2d_final,\n        stats2d_combine,\n        stats2d_trans_serialize,\n        stats2d_trans_deserialize\n    ],\n);\n\n//  Currently, rollup does not have the inverse function so if you want the behavior where we don't use the inverse,\n// you can use it in your window functions (useful for our own perf testing as well)\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(ss statssummary1d)\\n\\\n    (\\n\\\n        sfunc = stats1d_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats1d_final,\\n\\\n        combinefunc = stats1d_combine,\\n\\\n        serialfunc = stats1d_trans_serialize,\\n\\\n        deserialfunc = stats1d_trans_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_1d_rollup\",\n    requires = [\n        stats1d_summary_trans,\n        stats1d_final,\n        stats1d_combine,\n        stats1d_trans_serialize,\n        stats1d_trans_deserialize\n    ],\n);\n\n//  For UI, we decided to have slightly differently named functions for the windowed context and not, so that it reads better, as well as using the inverse function only in the window context\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rolling(ss statssummary1d)\\n\\\n    (\\n\\\n        sfunc = stats1d_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats1d_final,\\n\\\n        combinefunc = stats1d_combine,\\n\\\n        serialfunc = stats1d_trans_serialize,\\n\\\n        deserialfunc = stats1d_trans_deserialize,\\n\\\n        msfunc = stats1d_summary_trans,\\n\\\n        minvfunc = stats1d_summary_inv_trans,\\n\\\n        mstype = internal,\\n\\\n        mfinalfunc = stats1d_final,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_1d_rolling\",\n    requires = [\n        stats1d_summary_trans,\n        stats1d_final,\n        stats1d_combine,\n        stats1d_trans_serialize,\n        stats1d_trans_deserialize,\n        stats1d_summary_inv_trans\n    ],\n);\n\n// Same as for the 1D case, but for the 2D\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(ss statssummary2d)\\n\\\n    (\\n\\\n        sfunc = stats2d_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats2d_final,\\n\\\n        combinefunc = stats2d_combine,\\n\\\n        serialfunc = stats2d_trans_serialize,\\n\\\n        deserialfunc = stats2d_trans_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_2d_rollup\",\n    requires = [\n        stats2d_summary_trans,\n        stats2d_final,\n        stats2d_combine,\n        stats2d_trans_serialize,\n        stats2d_trans_deserialize\n    ],\n);\n\n//  For UI, we decided to have slightly differently named functions for the windowed context and not, so that it reads better, as well as using the inverse function only in the window context\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rolling(ss statssummary2d)\\n\\\n    (\\n\\\n        sfunc = stats2d_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = stats2d_final,\\n\\\n        combinefunc = stats2d_combine,\\n\\\n        serialfunc = stats2d_trans_serialize,\\n\\\n        deserialfunc = stats2d_trans_deserialize,\\n\\\n        msfunc = stats2d_summary_trans,\\n\\\n        minvfunc = stats2d_summary_inv_trans,\\n\\\n        mstype = internal,\\n\\\n        mfinalfunc = stats2d_final,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"stats_2d_rolling\",\n    requires = [\n        stats2d_summary_trans,\n        stats2d_final,\n        stats2d_combine,\n        stats2d_trans_serialize,\n        stats2d_trans_deserialize,\n        stats2d_summary_inv_trans\n    ],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_average(sketch: StatsSummary1D, _accessor: AccessorAverage) -> Option<f64> {\n    stats1d_average(sketch)\n}\n\n#[pg_extern(name = \"average\", strict, immutable, parallel_safe)]\npub(crate) fn stats1d_average(summary: StatsSummary1D) -> Option<f64> {\n    summary.to_internal().avg()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_sum(sketch: StatsSummary1D, _accessor: AccessorSum) -> Option<f64> {\n    stats1d_sum(sketch)\n}\n\n#[pg_extern(name = \"sum\", strict, immutable, parallel_safe)]\npub(crate) fn stats1d_sum(summary: StatsSummary1D) -> Option<f64> {\n    summary.to_internal().sum()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_stddev(\n    sketch: Option<StatsSummary1D>,\n    accessor: AccessorStdDev,\n) -> Option<f64> {\n    stats1d_stddev(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"stddev\", immutable, parallel_safe)]\nfn stats1d_stddev(\n    summary: Option<StatsSummary1D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => summary?.to_internal().stddev_pop(),\n        Sample => summary?.to_internal().stddev_samp(),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_variance(\n    sketch: Option<StatsSummary1D>,\n    accessor: AccessorVariance,\n) -> Option<f64> {\n    stats1d_variance(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"variance\", immutable, parallel_safe)]\nfn stats1d_variance(\n    summary: Option<StatsSummary1D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => summary?.to_internal().var_pop(),\n        Sample => summary?.to_internal().var_samp(),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_skewness(sketch: StatsSummary1D, accessor: AccessorSkewness) -> Option<f64> {\n    stats1d_skewness(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"skewness\", immutable, parallel_safe)]\nfn stats1d_skewness(summary: StatsSummary1D, method: default!(&str, \"'sample'\")) -> Option<f64> {\n    match method_kind(method) {\n        Population => summary.to_internal().skewness_pop(),\n        Sample => summary.to_internal().skewness_samp(),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_kurtosis(sketch: StatsSummary1D, accessor: AccessorKurtosis) -> Option<f64> {\n    stats1d_kurtosis(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"kurtosis\", immutable, parallel_safe)]\nfn stats1d_kurtosis(summary: StatsSummary1D, method: default!(&str, \"'sample'\")) -> Option<f64> {\n    match method_kind(method) {\n        Population => summary.to_internal().kurtosis_pop(),\n        Sample => summary.to_internal().kurtosis_samp(),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats1d_num_vals(sketch: StatsSummary1D, _accessor: AccessorNumVals) -> i64 {\n    stats1d_num_vals(sketch)\n}\n\n#[pg_extern(name = \"num_vals\", strict, immutable, parallel_safe)]\nfn stats1d_num_vals(summary: StatsSummary1D) -> i64 {\n    summary.to_internal().count()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_average_x(sketch: StatsSummary2D, _accessor: AccessorAverageX) -> Option<f64> {\n    stats2d_average_x(sketch)\n}\n\n#[pg_extern(name = \"average_x\", strict, immutable, parallel_safe)]\nfn stats2d_average_x(summary: StatsSummary2D) -> Option<f64> {\n    Some(summary.to_internal().avg()?.x)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_average_y(sketch: StatsSummary2D, _accessor: AccessorAverageY) -> Option<f64> {\n    stats2d_average_y(sketch)\n}\n\n#[pg_extern(name = \"average_y\", strict, immutable, parallel_safe)]\nfn stats2d_average_y(summary: StatsSummary2D) -> Option<f64> {\n    Some(summary.to_internal().avg()?.y)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_sum_x(sketch: StatsSummary2D, _accessor: AccessorSumX) -> Option<f64> {\n    stats2d_sum_x(sketch)\n}\n\n#[pg_extern(name = \"sum_x\", strict, immutable, parallel_safe)]\nfn stats2d_sum_x(summary: StatsSummary2D) -> Option<f64> {\n    Some(summary.to_internal().sum()?.x)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_sum_y(sketch: StatsSummary2D, _accessor: AccessorSumY) -> Option<f64> {\n    stats2d_sum_y(sketch)\n}\n\n#[pg_extern(name = \"sum_y\", strict, immutable, parallel_safe)]\nfn stats2d_sum_y(summary: StatsSummary2D) -> Option<f64> {\n    Some(summary.to_internal().sum()?.y)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_stdddev_x(\n    sketch: Option<StatsSummary2D>,\n    accessor: AccessorStdDevX,\n) -> Option<f64> {\n    stats2d_stddev_x(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"stddev_x\", immutable, parallel_safe)]\nfn stats2d_stddev_x(\n    summary: Option<StatsSummary2D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary?.to_internal().stddev_pop()?.x),\n        Sample => Some(summary?.to_internal().stddev_samp()?.x),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_stdddev_y(\n    sketch: Option<StatsSummary2D>,\n    accessor: AccessorStdDevY,\n) -> Option<f64> {\n    stats2d_stddev_y(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"stddev_y\", immutable, parallel_safe)]\nfn stats2d_stddev_y(\n    summary: Option<StatsSummary2D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary?.to_internal().stddev_pop()?.y),\n        Sample => Some(summary?.to_internal().stddev_samp()?.y),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_variance_x(\n    sketch: Option<StatsSummary2D>,\n    accessor: AccessorVarianceX,\n) -> Option<f64> {\n    stats2d_variance_x(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"variance_x\", immutable, parallel_safe)]\nfn stats2d_variance_x(\n    summary: Option<StatsSummary2D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary?.to_internal().var_pop()?.x),\n        Sample => Some(summary?.to_internal().var_samp()?.x),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_variance_y(\n    sketch: Option<StatsSummary2D>,\n    accessor: AccessorVarianceY,\n) -> Option<f64> {\n    stats2d_variance_y(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"variance_y\", immutable, parallel_safe)]\nfn stats2d_variance_y(\n    summary: Option<StatsSummary2D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary?.to_internal().var_pop()?.y),\n        Sample => Some(summary?.to_internal().var_samp()?.y),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_skewness_x(\n    sketch: StatsSummary2D,\n    accessor: AccessorSkewnessX,\n) -> Option<f64> {\n    stats2d_skewness_x(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"skewness_x\", strict, immutable, parallel_safe)]\nfn stats2d_skewness_x(summary: StatsSummary2D, method: default!(&str, \"'sample'\")) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary.to_internal().skewness_pop()?.x),\n        Sample => Some(summary.to_internal().skewness_samp()?.x),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_skewness_y(\n    sketch: StatsSummary2D,\n    accessor: AccessorSkewnessY,\n) -> Option<f64> {\n    stats2d_skewness_y(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"skewness_y\", strict, immutable, parallel_safe)]\nfn stats2d_skewness_y(summary: StatsSummary2D, method: default!(&str, \"'sample'\")) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary.to_internal().skewness_pop()?.y),\n        Sample => Some(summary.to_internal().skewness_samp()?.y),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_kurtosis_x(\n    sketch: StatsSummary2D,\n    accessor: AccessorKurtosisX,\n) -> Option<f64> {\n    stats2d_kurtosis_x(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"kurtosis_x\", strict, immutable, parallel_safe)]\nfn stats2d_kurtosis_x(summary: StatsSummary2D, method: default!(&str, \"'sample'\")) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary.to_internal().kurtosis_pop()?.x),\n        Sample => Some(summary.to_internal().kurtosis_samp()?.x),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_kurtosis_y(\n    sketch: StatsSummary2D,\n    accessor: AccessorKurtosisY,\n) -> Option<f64> {\n    stats2d_kurtosis_y(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"kurtosis_y\", strict, immutable, parallel_safe)]\nfn stats2d_kurtosis_y(summary: StatsSummary2D, method: default!(&str, \"'sample'\")) -> Option<f64> {\n    match method_kind(method) {\n        Population => Some(summary.to_internal().kurtosis_pop()?.y),\n        Sample => Some(summary.to_internal().kurtosis_samp()?.y),\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_num_vals(sketch: StatsSummary2D, _accessor: AccessorNumVals) -> i64 {\n    stats2d_num_vals(sketch)\n}\n\n#[pg_extern(name = \"num_vals\", strict, immutable, parallel_safe)]\nfn stats2d_num_vals(summary: StatsSummary2D) -> i64 {\n    summary.to_internal().count()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_slope(sketch: StatsSummary2D, _accessor: AccessorSlope) -> Option<f64> {\n    stats2d_slope(sketch)\n}\n\n#[pg_extern(name = \"slope\", strict, immutable, parallel_safe)]\nfn stats2d_slope(summary: StatsSummary2D) -> Option<f64> {\n    summary.to_internal().slope()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_corr(sketch: StatsSummary2D, _accessor: AccessorCorr) -> Option<f64> {\n    stats2d_corr(sketch)\n}\n\n#[pg_extern(name = \"corr\", strict, immutable, parallel_safe)]\nfn stats2d_corr(summary: StatsSummary2D) -> Option<f64> {\n    summary.to_internal().corr()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_intercept(\n    sketch: StatsSummary2D,\n    _accessor: AccessorIntercept,\n) -> Option<f64> {\n    stats2d_intercept(sketch)\n}\n\n#[pg_extern(name = \"intercept\", strict, immutable, parallel_safe)]\nfn stats2d_intercept(summary: StatsSummary2D) -> Option<f64> {\n    summary.to_internal().intercept()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_x_intercept(\n    sketch: StatsSummary2D,\n    _accessor: AccessorXIntercept,\n) -> Option<f64> {\n    stats2d_x_intercept(sketch)\n}\n\n#[pg_extern(name = \"x_intercept\", strict, immutable, parallel_safe)]\nfn stats2d_x_intercept(summary: StatsSummary2D) -> Option<f64> {\n    summary.to_internal().x_intercept()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_determination_coeff(\n    sketch: StatsSummary2D,\n    _accessor: AccessorDeterminationCoeff,\n) -> Option<f64> {\n    stats2d_determination_coeff(sketch)\n}\n\n#[pg_extern(name = \"determination_coeff\", strict, immutable, parallel_safe)]\nfn stats2d_determination_coeff(summary: StatsSummary2D) -> Option<f64> {\n    summary.to_internal().determination_coeff()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_stats2d_covar(sketch: Option<StatsSummary2D>, accessor: AccessorCovar) -> Option<f64> {\n    stats2d_covar(sketch, accessor.method.as_str())\n}\n\n#[pg_extern(name = \"covariance\", immutable, parallel_safe)]\nfn stats2d_covar(\n    summary: Option<StatsSummary2D>,\n    method: default!(&str, \"'sample'\"),\n) -> Option<f64> {\n    match method_kind(method) {\n        Population => summary?.to_internal().covar_pop(),\n        Sample => summary?.to_internal().covar_samp(),\n    }\n}\n\n#[derive(\n    Clone, Copy, Debug, serde::Serialize, serde::Deserialize, flat_serialize_macro::FlatSerializable,\n)]\n#[repr(u8)]\npub enum Method {\n    Population = 1,\n    Sample = 2,\n}\n\nimpl Method {\n    pub fn as_str(&self) -> &'static str {\n        match self {\n            Population => \"population\",\n            Sample => \"sample\",\n        }\n    }\n}\n\n#[track_caller]\npub fn method_kind(method: &str) -> Method {\n    match as_method(method) {\n        Some(method) => method,\n        None => {\n            pgrx::error!(\"unknown analysis method. Valid methods are 'population' and 'sample'\")\n        }\n    }\n}\n\npub fn as_method(method: &str) -> Option<Method> {\n    match method.trim().to_lowercase().as_str() {\n        \"population\" | \"pop\" => Some(Population),\n        \"sample\" | \"samp\" => Some(Sample),\n        _ => None,\n    }\n}\n\n// TODO: Add testing - probably want to do some fuzz testing against the Postgres implementations of the same. Possibly translate the Postgres tests as well?\n// #[cfg(any(test, feature = \"pg_test\"))]\n// mod tests {\n\n//     use approx::assert_relative_eq;\n//     use pgrx::*;\n//     use super::*;\n\n//     macro_rules! select_one {\n//         ($client:expr, $stmt:expr, $type:ty) => {\n//             $client\n//                 .update($stmt, None, &[])\n//                 .first()\n//                 .get_one::<$type>()\n//                 .unwrap()\n//                 .unwrap()\n//         };\n//     }\n\n//     //do proper numerical comparisons on the values where that matters, use exact where it should be exact.\n//     #[track_caller]\n//     fn stats1d_assert_close_enough(p1:&StatsSummary1D, p2:&StatsSummary1D) {\n//         assert_eq!(p1.n, p2.n, \"n\");\n//         assert_relative_eq!(p1.sx, p2.sx);\n//         assert_relative_eq!(p1.sxx, p2.sxx);\n//     }\n//     #[track_caller]\n//     fn stats2d_assert_close_enough(p1:&StatsSummary2D, p2:&StatsSummary2D) {\n//         assert_eq!(p1.n, p2.n, \"n\");\n//         assert_relative_eq!(p1.sx, p2.sx);\n//         assert_relative_eq!(p1.sxx, p2.sxx);\n//         assert_relative_eq!(p1.sy, p2.sy);\n//         assert_relative_eq!(p1.syy, p2.syy);\n//         assert_relative_eq!(p1.sxy, p2.sxy);\n//     }\n\n//     // #[pg_test]\n//     // fn test_combine_aggregate(){\n//     //     Spi::connect_mut(|client| {\n\n//     //     });\n//     // }\n// }\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n    use approx::relative_eq;\n\n    use pgrx_macros::pg_test;\n    use rand::rngs::SmallRng;\n    use rand::seq::SliceRandom;\n    use rand::{self, Rng, SeedableRng};\n\n    const RUNS: usize = 10; // Number of runs to generate\n    const VALS: usize = 10000; // Number of values to use for each run\n    const SEED: Option<u64> = None; // RNG seed, generated from entropy if None\n    const PRINT_VALS: bool = false; // Print out test values on error, this can be spammy if VALS is high\n\n    #[pg_test]\n    fn test_stats_agg_text_io() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE test_table (test_x DOUBLE PRECISION, test_y DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let test = client\n                .update(\n                    \"SELECT stats_agg(test_y, test_x)::TEXT FROM test_table\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert!(test.is_none());\n\n            client\n                .update(\"INSERT INTO test_table VALUES (10, 10);\", None, &[])\n                .unwrap();\n\n            let test = client\n                .update(\n                    \"SELECT stats_agg(test_y, test_x)::TEXT FROM test_table\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(\n                test,\n                \"(version:1,n:1,sx:10,sx2:0,sx3:0,sx4:0,sy:10,sy2:0,sy3:0,sy4:0,sxy:0)\"\n            );\n\n            client\n                .update(\"INSERT INTO test_table VALUES (20, 20);\", None, &[])\n                .unwrap();\n            let test = client\n                .update(\n                    \"SELECT stats_agg(test_y, test_x)::TEXT FROM test_table\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            let expected =\n                \"(version:1,n:2,sx:30,sx2:50,sx3:0,sx4:1250,sy:30,sy2:50,sy3:0,sy4:1250,sxy:50)\";\n            assert_eq!(test, expected);\n\n            // Test a few functions to see that the text serialized object behave the same as the constructed one\n            assert_eq!(\n                client\n                    .update(\n                        \"SELECT skewness_x(stats_agg(test_y, test_x)) FROM test_table\",\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>(),\n                client\n                    .update(\n                        &format!(\"SELECT skewness_x('{expected}'::StatsSummary2D)\"),\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>()\n            );\n            assert_eq!(\n                client\n                    .update(\n                        \"SELECT kurtosis_y(stats_agg(test_y, test_x)) FROM test_table\",\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>(),\n                client\n                    .update(\n                        &format!(\"SELECT kurtosis_y('{expected}'::StatsSummary2D)\"),\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>()\n            );\n            assert_eq!(\n                client\n                    .update(\n                        \"SELECT covariance(stats_agg(test_y, test_x)) FROM test_table\",\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>(),\n                client\n                    .update(\n                        &format!(\"SELECT covariance('{expected}'::StatsSummary2D)\"),\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>()\n            );\n\n            // Test text round trip\n            assert_eq!(\n                client\n                    .update(\n                        &format!(\"SELECT '{expected}'::StatsSummary2D::TEXT\"),\n                        None,\n                        &[]\n                    )\n                    .unwrap()\n                    .first()\n                    .get_one::<String>()\n                    .unwrap()\n                    .unwrap(),\n                expected\n            );\n\n            client\n                .update(\"INSERT INTO test_table VALUES ('NaN', 30);\", None, &[])\n                .unwrap();\n            let test = client\n                .update(\n                    \"SELECT stats_agg(test_y, test_x)::TEXT FROM test_table\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test, \"(version:1,n:3,sx:NaN,sx2:NaN,sx3:NaN,sx4:NaN,sy:60,sy2:200,sy3:0,sy4:20000,sxy:NaN)\");\n\n            client\n                .update(\"INSERT INTO test_table VALUES (40, 'Inf');\", None, &[])\n                .unwrap();\n            let test = client\n                .update(\n                    \"SELECT stats_agg(test_y, test_x)::TEXT FROM test_table\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test, \"(version:1,n:4,sx:NaN,sx2:NaN,sx3:NaN,sx4:NaN,sy:inf,sy2:NaN,sy3:NaN,sy4:NaN,sxy:NaN)\");\n        });\n    }\n\n    #[pg_test]\n    fn test_stats_agg_byte_io() {\n        unsafe {\n            use std::ptr;\n            let state = stats1d_trans_inner(None, Some(14.0), ptr::null_mut());\n            let state = stats1d_trans_inner(state, Some(18.0), ptr::null_mut());\n            let state = stats1d_trans_inner(state, Some(22.7), ptr::null_mut());\n            let state = stats1d_trans_inner(state, Some(39.42), ptr::null_mut());\n            let state = stats1d_trans_inner(state, Some(-43.0), ptr::null_mut());\n\n            let control = (*state.unwrap()).clone();\n            let buffer = stats1d_trans_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let expected = [\n                1, 1, 1, 5, 0, 0, 0, 0, 0, 0, 0, 144, 194, 245, 40, 92, 143, 73, 64, 100, 180, 142,\n                170, 38, 151, 174, 64, 72, 48, 180, 190, 189, 33, 254, 192, 119, 78, 30, 195, 209,\n                190, 96, 65,\n            ];\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let new_state =\n                stats1d_trans_deserialize_inner(bytea(pg_sys::Datum::from(expected.as_ptr())));\n\n            assert_eq!(*new_state, control);\n        }\n    }\n\n    #[pg_test]\n    fn stats_agg_fuzz() {\n        let mut state = TestState::new(RUNS, VALS, SEED);\n        for _ in 0..state.runs {\n            state.populate_values();\n            test_aggs(&mut state);\n            state.passed += 1;\n        }\n    }\n\n    struct TestState {\n        runs: usize,\n        values: usize,\n        passed: usize,\n        x_values: Vec<f64>,\n        y_values: Vec<f64>,\n        seed: u64,\n        r#gen: SmallRng,\n    }\n\n    impl TestState {\n        pub fn new(runs: usize, values: usize, seed: Option<u64>) -> TestState {\n            let seed = match seed {\n                Some(s) => s,\n                None => SmallRng::from_entropy().gen_range(0..u64::MAX),\n            };\n\n            TestState {\n                runs,\n                values,\n                passed: 0,\n                x_values: Vec::new(),\n                y_values: Vec::new(),\n                seed,\n                r#gen: SmallRng::seed_from_u64(seed),\n            }\n        }\n\n        pub fn populate_values(&mut self) {\n            // Discard old values\n            self.x_values = Vec::with_capacity(self.values);\n            self.y_values = Vec::with_capacity(self.values);\n\n            // We'll cluster the exponential components of the random values around a particular value\n            let exp_base = self\n                .r#gen\n                .gen_range((f64::MIN_EXP / 10) as f64..(f64::MAX_EXP / 10) as f64);\n\n            for _ in 0..self.values {\n                let exp = self.r#gen.gen_range((exp_base - 2.)..=(exp_base + 2.));\n                let mantissa = self.r#gen.gen_range((1.)..2.);\n                let sign = [-1., 1.].choose(&mut self.r#gen).unwrap();\n                self.x_values.push(sign * mantissa * exp.exp2());\n\n                let exp = self.r#gen.gen_range((exp_base - 2.)..=(exp_base + 2.));\n                let mantissa = self.r#gen.gen_range((1.)..2.);\n                let sign = [-1., 1.].choose(&mut self.r#gen).unwrap();\n                self.y_values.push(sign * mantissa * exp.exp2());\n            }\n        }\n\n        pub fn failed_msg(&self, dump_vals: bool) -> String {\n            format!(\"Failed after {} successful iterations, run using {} values generated from seed {}{}\", self.passed, self.x_values.len(), self.seed,\n                if dump_vals {\n                    format!(\"\\nX-values:\\n{:?}\\n\\nY-values:\\n{:?}\", self.x_values, self.y_values)\n                } else {\n                    \"\".to_string()\n                }\n            )\n        }\n    }\n\n    #[allow(clippy::float_cmp)]\n    fn check_agg_equivalence(\n        state: &TestState,\n        client: &mut pgrx::spi::SpiClient,\n        pg_cmd: &str,\n        tk_cmd: &str,\n        allowed_diff: f64,\n        do_moving_agg: bool,\n    ) {\n        warning!(\"pg_cmd={} ; tk_cmd={}\", pg_cmd, tk_cmd);\n        let pg_row = client.update(pg_cmd, None, &[]).unwrap().first();\n        let (pg_result, pg_moving_agg_result) = if do_moving_agg {\n            pg_row.get_two::<f64, f64>().unwrap()\n        } else {\n            (pg_row.get_one::<f64>().unwrap(), None)\n        };\n        let pg_result = pg_result.unwrap();\n\n        let (tk_result, arrow_result, tk_moving_agg_result) = client\n            .update(tk_cmd, None, &[])\n            .unwrap()\n            .first()\n            .get_three::<f64, f64, f64>()\n            .unwrap();\n        let (tk_result, arrow_result) = (tk_result.unwrap(), arrow_result.unwrap());\n        assert_eq!(tk_result, arrow_result, \"Arrow didn't match in {tk_cmd}\");\n\n        let result = if allowed_diff == 0.0 {\n            pg_result == tk_result\n        } else {\n            relative_eq!(pg_result, tk_result, max_relative = allowed_diff)\n        };\n\n        if !result {\n            let abs_diff = f64::abs(pg_result - tk_result);\n            let abs_max = f64::abs(pg_result).max(f64::abs(tk_result));\n            panic!(\n                \"Output didn't match between postgres command: {}\\n\\\n                and stats_agg command: {} \\n\\\n                \\tpostgres result: {}\\n\\\n                \\tstatsagg result: {}\\n\\\n                \\trelative difference:         {}\\n\\\n                \\tallowed relative difference: {}\\n\\\n                {}\",\n                pg_cmd,\n                tk_cmd,\n                pg_result,\n                tk_result,\n                abs_diff / abs_max,\n                allowed_diff,\n                state.failed_msg(PRINT_VALS)\n            );\n        }\n\n        if do_moving_agg {\n            approx::assert_relative_eq!(\n                pg_moving_agg_result.unwrap(),\n                tk_moving_agg_result.unwrap(),\n                max_relative = 1e-9,\n            )\n        }\n    }\n\n    fn pg1d_aggx(agg: &str) -> String {\n        format!(\"SELECT {agg}(test_x)::float, (SELECT {agg}(test_x) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3)::float FROM test_table\")\n    }\n\n    fn pg1d_aggy(agg: &str) -> String {\n        format!(\"SELECT {agg}(test_y), (SELECT {agg}(test_y) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3) FROM test_table\")\n    }\n\n    fn pg2d_agg(agg: &str) -> String {\n        format!(\"SELECT {agg}(test_y, test_x)::float, (SELECT {agg}(test_y, test_x) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3)::float FROM test_table\")\n    }\n\n    fn tk1d_agg(agg: &str) -> String {\n        format!(\n            \"SELECT \\\n            {agg}(stats_agg(test_x))::float, \\\n            (stats_agg(test_x)->{agg}())::float, \\\n            {agg}((SELECT stats_agg(test_x) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3))::float \\\n        FROM test_table\"\n        )\n    }\n\n    fn tk1d_agg_arg(agg: &str, arg: &str) -> String {\n        format!(\n            \"SELECT \\\n            {agg}(stats_agg(test_x), '{arg}'), \\\n            stats_agg(test_x)->{agg}('{arg}'), \\\n            {agg}((SELECT stats_agg(test_x) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3), '{arg}') \\\n        FROM test_table\"\n        )\n    }\n\n    fn tk2d_agg(agg: &str) -> String {\n        format!(\n            \"SELECT \\\n            {agg}(stats_agg(test_y, test_x))::float, \\\n            (stats_agg(test_y, test_x)->{agg}())::float, \\\n            {agg}((SELECT stats_agg(test_y, test_x) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3))::float \\\n        FROM test_table\"\n        )\n    }\n\n    fn tk2d_agg_arg(agg: &str, arg: &str) -> String {\n        format!(\n            \"SELECT \\\n            {agg}(stats_agg(test_y, test_x), '{arg}'), \\\n            stats_agg(test_y, test_x)->{agg}('{arg}'), \\\n            {agg}((SELECT stats_agg(test_y, test_x) OVER (ORDER BY test_x ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) FROM test_table LIMIT 1 OFFSET 3), '{arg}') \\\n        FROM test_table\"\n        )\n    }\n\n    fn pg_moment_pop_query(moment: i32, column: &str) -> String {\n        format!(\"select sum(({column} - a.avg)^{moment}) / count({column}) / (stddev_pop({column})^{moment}) from test_table, (select avg({column}) from test_table) a\")\n    }\n\n    fn pg_moment_samp_query(moment: i32, column: &str) -> String {\n        format!(\"select sum(({column} - a.avg)^{moment}) / (count({column}) - 1) / (stddev_samp({column})^{moment}) from test_table, (select avg({column}) from test_table) a\")\n    }\n\n    fn test_aggs(state: &mut TestState) {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE test_table (test_x DOUBLE PRECISION, test_y DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    &format!(\n                        \"INSERT INTO test_table VALUES {}\",\n                        state\n                            .x_values\n                            .iter()\n                            .zip(state.y_values.iter())\n                            .map(|(x, y)| \"(\".to_string()\n                                + &x.to_string()\n                                + \",\"\n                                + &y.to_string()\n                                + \")\"\n                                + \",\")\n                            .collect::<String>()\n                            .trim_end_matches(',')\n                    ),\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Definitions for allowed errors for different aggregates\n            const NONE: f64 = 0.; // Exact match\n            const EPS1: f64 = f64::EPSILON; // Generally enough to handle float rounding\n            const EPS2: f64 = 2. * f64::EPSILON; // stddev is sqrt(variance), so a bit looser bound\n            const EPS3: f64 = 3. * f64::EPSILON; // Sum of squares in variance agg accumulates a bit more error\n            const BILLIONTH: f64 = 1e-9; // Higher order moments exponentially compound the error\n\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"avg\"),\n                &tk1d_agg(\"average\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"sum\"),\n                &tk1d_agg(\"sum\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"count\"),\n                &tk1d_agg(\"num_vals\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"stddev\"),\n                &tk1d_agg(\"stddev\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"stddev_pop\"),\n                &tk1d_agg_arg(\"stddev\", \"population\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"stddev_samp\"),\n                &tk1d_agg_arg(\"stddev\", \"sample\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"variance\"),\n                &tk1d_agg(\"variance\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"var_pop\"),\n                &tk1d_agg_arg(\"variance\", \"population\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"var_samp\"),\n                &tk1d_agg_arg(\"variance\", \"sample\"),\n                EPS3,\n                true,\n            );\n\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"regr_avgx\"),\n                &tk2d_agg(\"average_x\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"regr_avgy\"),\n                &tk2d_agg(\"average_y\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"sum\"),\n                &tk2d_agg(\"sum_x\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"sum\"),\n                &tk2d_agg(\"sum_y\"),\n                NONE,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"stddev\"),\n                &tk2d_agg(\"stddev_x\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"stddev\"),\n                &tk2d_agg(\"stddev_y\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"stddev_pop\"),\n                &tk2d_agg_arg(\"stddev_x\", \"population\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"stddev_pop\"),\n                &tk2d_agg_arg(\"stddev_y\", \"population\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"stddev_samp\"),\n                &tk2d_agg_arg(\"stddev_x\", \"sample\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"stddev_samp\"),\n                &tk2d_agg_arg(\"stddev_y\", \"sample\"),\n                EPS2,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"variance\"),\n                &tk2d_agg(\"variance_x\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"variance\"),\n                &tk2d_agg(\"variance_y\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"var_pop\"),\n                &tk2d_agg_arg(\"variance_x\", \"population\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"var_pop\"),\n                &tk2d_agg_arg(\"variance_y\", \"population\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggx(\"var_samp\"),\n                &tk2d_agg_arg(\"variance_x\", \"sample\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg1d_aggy(\"var_samp\"),\n                &tk2d_agg_arg(\"variance_y\", \"sample\"),\n                EPS3,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"regr_count\"),\n                &tk2d_agg(\"num_vals\"),\n                NONE,\n                true,\n            );\n\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"regr_slope\"),\n                &tk2d_agg(\"slope\"),\n                EPS1,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"corr\"),\n                &tk2d_agg(\"corr\"),\n                EPS1,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"regr_intercept\"),\n                &tk2d_agg(\"intercept\"),\n                EPS1,\n                true,\n            );\n\n            // No postgres equivalent for x_intercept, so we only test function vs. arrow operator.\n            {\n                let query = tk2d_agg(\"x_intercept\");\n                let (result, arrow_result) = client\n                    .update(&query, None, &[])\n                    .unwrap()\n                    .first()\n                    .get_two::<f64, f64>()\n                    .unwrap();\n                assert_eq!(result, arrow_result, \"Arrow didn't match in {query}\");\n            }\n\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"regr_r2\"),\n                &tk2d_agg(\"determination_coeff\"),\n                EPS1,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"covar_pop\"),\n                &tk2d_agg_arg(\"covariance\", \"population\"),\n                BILLIONTH,\n                true,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg2d_agg(\"covar_samp\"),\n                &tk2d_agg_arg(\"covariance\", \"sample\"),\n                BILLIONTH,\n                true,\n            );\n\n            // Skewness and kurtosis don't have aggregate functions in postgres, but we can compute them\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_pop_query(3, \"test_x\"),\n                &tk1d_agg_arg(\"skewness\", \"population\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_pop_query(3, \"test_x\"),\n                &tk2d_agg_arg(\"skewness_x\", \"population\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_pop_query(3, \"test_y\"),\n                &tk2d_agg_arg(\"skewness_y\", \"population\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_pop_query(4, \"test_x\"),\n                &tk1d_agg_arg(\"kurtosis\", \"population\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_pop_query(4, \"test_x\"),\n                &tk2d_agg_arg(\"kurtosis_x\", \"population\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_pop_query(4, \"test_y\"),\n                &tk2d_agg_arg(\"kurtosis_y\", \"population\"),\n                BILLIONTH,\n                false,\n            );\n\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_samp_query(3, \"test_x\"),\n                &tk1d_agg_arg(\"skewness\", \"sample\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_samp_query(3, \"test_x\"),\n                &tk2d_agg_arg(\"skewness_x\", \"sample\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_samp_query(3, \"test_y\"),\n                &tk2d_agg_arg(\"skewness_y\", \"sample\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_samp_query(4, \"test_x\"),\n                &tk1d_agg_arg(\"kurtosis\", \"sample\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_samp_query(4, \"test_x\"),\n                &tk2d_agg_arg(\"kurtosis_x\", \"sample\"),\n                BILLIONTH,\n                false,\n            );\n            check_agg_equivalence(\n                state,\n                client,\n                &pg_moment_samp_query(4, \"test_y\"),\n                &tk2d_agg_arg(\"kurtosis_y\", \"sample\"),\n                BILLIONTH,\n                false,\n            );\n\n            client.update(\"DROP TABLE test_table\", None, &[]).unwrap();\n        });\n    }\n\n    #[pg_test]\n    fn stats_agg_rolling() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"\nSET timezone TO 'UTC';\nCREATE TABLE prices(ts TIMESTAMPTZ, price FLOAT);\nINSERT INTO prices (\n    WITH dates AS\n        (SELECT\n            *\n        FROM\n            generate_series('2020-01-01 00:00'::timestamp, '2020-02-01 12:00', '10 minutes') time)\n    SELECT\n        dates.time,\n        (select (random()+EXTRACT(seconds FROM dates.time))*100 ) price\n    FROM\n        dates\n);\n\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut vals = client.update(\n                \"SELECT stddev(data.stats_agg) FROM (SELECT stats_agg(price) OVER (ORDER BY ts RANGE '50 minutes' PRECEDING) FROM prices) data\",\n                None, &[]\n            ).unwrap();\n            assert!(vals.next().unwrap()[1]\n                .value::<f64>()\n                .unwrap()\n                .unwrap()\n                .is_nan());\n            assert!(vals.next().unwrap()[1].value::<f64>().unwrap().is_some());\n            assert!(vals.next().unwrap()[1].value::<f64>().unwrap().is_some());\n\n            let mut vals = client.update(\n                \"SELECT slope(data.stats_agg) FROM (SELECT stats_agg((EXTRACT(minutes FROM ts)), price) OVER (ORDER BY ts RANGE '50 minutes' PRECEDING) FROM prices) data;\",\n                None, &[]\n            ).unwrap();\n            assert!(vals.next().unwrap()[1].value::<f64>().unwrap().is_none()); // trendline is zero initially\n            assert!(vals.next().unwrap()[1].value::<f64>().unwrap().is_some());\n            assert!(vals.next().unwrap()[1].value::<f64>().unwrap().is_some());\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/tdigest.rs",
    "content": "use std::{convert::TryInto, ops::Deref};\n\nuse pgrx::*;\n\nuse crate::{\n    accessors::{\n        AccessorApproxPercentile, AccessorApproxPercentileRank, AccessorMaxVal, AccessorMean,\n        AccessorMinVal, AccessorNumVals,\n    },\n    aggregate_utils::in_aggregate_context,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n};\n\nuse tdigest::{Centroid, TDigest as InternalTDigest};\n\n// PG function for adding values to a digest.\n// Null values are ignored.\n#[pg_extern(immutable, parallel_safe)]\npub fn tdigest_trans(\n    state: Internal,\n    size: i32,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    tdigest_trans_inner(unsafe { state.to_inner() }, size, value, fcinfo).internal()\n}\npub fn tdigest_trans_inner(\n    state: Option<Inner<tdigest::Builder>>,\n    size: i32,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<tdigest::Builder>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let value = match value {\n                None => return state,\n                // NaNs are nonsensical in the context of a percentile, so exclude them\n                Some(value) => {\n                    if value.is_nan() {\n                        return state;\n                    } else {\n                        value\n                    }\n                }\n            };\n            let mut state = match state {\n                None => tdigest::Builder::with_size(size.try_into().unwrap()).into(),\n                Some(state) => state,\n            };\n            state.push(value);\n            Some(state)\n        })\n    }\n}\n\n// PG function for merging digests.\n#[pg_extern(immutable, parallel_safe)]\npub fn tdigest_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { tdigest_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\n\npub fn tdigest_combine_inner(\n    state1: Option<Inner<tdigest::Builder>>,\n    state2: Option<Inner<tdigest::Builder>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<tdigest::Builder>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(state2)) => Some(state2.clone().into()),\n            (Some(state1), None) => Some(state1.clone().into()),\n            (Some(state1), Some(state2)) => {\n                let mut merged = state1.clone();\n                merged.merge(state2.clone());\n                Some(merged.into())\n            }\n        })\n    }\n}\n\nuse crate::raw::bytea;\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn tdigest_serialize(state: Internal) -> bytea {\n    let mut state = state;\n    let state: &mut tdigest::Builder = unsafe { state.get_mut().unwrap() };\n    // TODO this macro is really broken\n    let hack = state.build();\n    let hackref = &hack;\n    crate::do_serialize!(hackref)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn tdigest_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    tdigest_deserialize_inner(bytes).internal()\n}\npub fn tdigest_deserialize_inner(bytes: bytea) -> Inner<tdigest::Builder> {\n    crate::do_deserialize!(bytes, tdigest::Builder)\n}\n\n// PG object for the digest.\npg_type! {\n    #[derive(Debug)]\n    struct TDigest<'input> {\n        // We compute this.  It's a (harmless) bug that we serialize it.\n        #[serde(skip_deserializing)]\n        buckets: u32,\n        max_buckets: u32,\n        count: u64,\n        sum: f64,\n        min: f64,\n        max: f64,\n        centroids: [Centroid; self.buckets],\n    }\n}\n\nimpl<'input> InOutFuncs for TDigest<'input> {\n    fn output(&self, buffer: &mut StringInfo) {\n        use crate::serialization::{str_to_db_encoding, EncodedStr::*};\n\n        let stringified = ron::to_string(&**self).unwrap();\n        match str_to_db_encoding(&stringified) {\n            Utf8(s) => buffer.push_str(s),\n            Other(s) => buffer.push_bytes(s.to_bytes()),\n        }\n    }\n\n    fn input(input: &std::ffi::CStr) -> TDigest<'input>\n    where\n        Self: Sized,\n    {\n        use crate::serialization::str_from_db_encoding;\n\n        let input = str_from_db_encoding(input);\n        let mut val: TDigestData = ron::from_str(input).unwrap();\n        val.buckets = val\n            .centroids\n            .len()\n            .try_into()\n            .expect(\"centroids len fits into u32\");\n        unsafe { Self(val, crate::type_builder::CachedDatum::None).flatten() }\n    }\n}\n\nimpl<'input> TDigest<'input> {\n    fn to_internal_tdigest(&self) -> InternalTDigest {\n        InternalTDigest::new(\n            self.centroids.iter().collect(),\n            self.sum,\n            self.count,\n            self.max,\n            self.0.min,\n            self.max_buckets as usize,\n        )\n    }\n\n    fn from_internal_tdigest(digest: &InternalTDigest) -> TDigest<'static> {\n        let max_buckets: u32 = digest.max_size().try_into().unwrap();\n\n        let centroids = digest.raw_centroids();\n\n        // we need to flatten the vector to a single buffer that contains\n        // both the size, the data, and the varlen header\n        unsafe {\n            flatten!(TDigest {\n                max_buckets,\n                buckets: centroids.len() as u32,\n                count: digest.count(),\n                sum: digest.sum(),\n                min: digest.min(),\n                max: digest.max(),\n                centroids: centroids.into(),\n            })\n        }\n    }\n}\n\n// PG function to generate a user-facing TDigest object from an internal tdigest::Builder.\n#[pg_extern(immutable, parallel_safe)]\nfn tdigest_final(state: Internal, fcinfo: pg_sys::FunctionCallInfo) -> Option<TDigest<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = state;\n            let state: &mut tdigest::Builder = match state.get_mut() {\n                None => return None,\n                Some(state) => state,\n            };\n            TDigest::from_internal_tdigest(&state.build()).into()\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE tdigest(size integer, value DOUBLE PRECISION)\\n\\\n    (\\n\\\n        sfunc = tdigest_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = tdigest_final,\\n\\\n        combinefunc = tdigest_combine,\\n\\\n        serialfunc = tdigest_serialize,\\n\\\n        deserialfunc = tdigest_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"tdigest_agg\",\n    requires = [\n        tdigest_trans,\n        tdigest_final,\n        tdigest_combine,\n        tdigest_serialize,\n        tdigest_deserialize\n    ],\n);\n\n#[pg_extern(immutable, parallel_safe)]\npub fn tdigest_compound_trans(\n    state: Internal,\n    value: Option<TDigest<'static>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    tdigest_compound_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\npub fn tdigest_compound_trans_inner(\n    state: Option<Inner<InternalTDigest>>,\n    value: Option<TDigest<'static>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<InternalTDigest>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state, value) {\n                (a, None) => a,\n                (None, Some(a)) => Some(a.to_internal_tdigest().into()),\n                (Some(a), Some(b)) => {\n                    assert_eq!(a.max_size(), b.max_buckets as usize);\n                    Some(\n                        InternalTDigest::merge_digests(\n                            vec![a.deref().clone(), b.to_internal_tdigest()], // TODO: TDigest merge with self\n                        )\n                        .into(),\n                    )\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn tdigest_compound_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe {\n        tdigest_compound_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal()\n    }\n}\npub fn tdigest_compound_combine_inner(\n    state1: Option<Inner<InternalTDigest>>,\n    state2: Option<Inner<InternalTDigest>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<InternalTDigest>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state1, state2) {\n                (None, None) => None,\n                (None, Some(state2)) => Some(state2.clone().into()),\n                (Some(state1), None) => Some(state1.clone().into()),\n                (Some(state1), Some(state2)) => {\n                    assert_eq!(state1.max_size(), state2.max_size());\n                    Some(\n                        InternalTDigest::merge_digests(\n                            vec![state1.deref().clone(), state2.deref().clone()], // TODO: TDigest merge with self\n                        )\n                        .into(),\n                    )\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn tdigest_compound_final(\n    state: Internal,\n    _fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<TDigest<'static>> {\n    let state: Option<&InternalTDigest> = unsafe { state.get() };\n    state.map(TDigest::from_internal_tdigest)\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn tdigest_compound_serialize(state: Internal, _fcinfo: pg_sys::FunctionCallInfo) -> bytea {\n    let state: Inner<InternalTDigest> = unsafe { state.to_inner().unwrap() };\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn tdigest_compound_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let i: InternalTDigest = crate::do_deserialize!(bytes, InternalTDigest);\n    Inner::from(i).internal()\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        tdigest\\n\\\n    ) (\\n\\\n        sfunc = tdigest_compound_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = tdigest_compound_final,\\n\\\n        combinefunc = tdigest_compound_combine,\\n\\\n        serialfunc = tdigest_compound_serialize,\\n\\\n        deserialfunc = tdigest_compound_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"tdigest_rollup\",\n    requires = [\n        tdigest_compound_trans,\n        tdigest_compound_final,\n        tdigest_compound_combine,\n        tdigest_compound_serialize,\n        tdigest_compound_deserialize\n    ],\n);\n\n//---- Available PG operations on the digest\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_tdigest_approx_percentile<'a>(\n    sketch: TDigest<'a>,\n    accessor: AccessorApproxPercentile,\n) -> f64 {\n    tdigest_quantile(accessor.percentile, sketch)\n}\n\n// Approximate the value at the given quantile (0.0-1.0)\n#[pg_extern(immutable, parallel_safe, name = \"approx_percentile\")]\npub fn tdigest_quantile<'a>(quantile: f64, digest: TDigest<'a>) -> f64 {\n    digest.to_internal_tdigest().estimate_quantile(quantile)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_tdigest_approx_rank<'a>(\n    sketch: TDigest<'a>,\n    accessor: AccessorApproxPercentileRank,\n) -> f64 {\n    tdigest_quantile_at_value(accessor.value, sketch)\n}\n\n// Approximate the quantile at the given value\n#[pg_extern(immutable, parallel_safe, name = \"approx_percentile_rank\")]\npub fn tdigest_quantile_at_value<'a>(value: f64, digest: TDigest<'a>) -> f64 {\n    digest\n        .to_internal_tdigest()\n        .estimate_quantile_at_value(value)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_tdigest_num_vals<'a>(sketch: TDigest<'a>, _accessor: AccessorNumVals) -> f64 {\n    tdigest_count(sketch)\n}\n\n// Number of elements from which the digest was built.\n#[pg_extern(immutable, parallel_safe, name = \"num_vals\")]\npub fn tdigest_count<'a>(digest: TDigest<'a>) -> f64 {\n    digest.count as f64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_tdigest_min<'a>(sketch: TDigest<'a>, _accessor: AccessorMinVal) -> f64 {\n    tdigest_min(sketch)\n}\n\n// Minimum value entered in the digest.\n#[pg_extern(immutable, parallel_safe, name = \"min_val\")]\npub fn tdigest_min<'a>(digest: TDigest<'a>) -> f64 {\n    digest.min\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_tdigest_max<'a>(sketch: TDigest<'a>, _accessor: AccessorMaxVal) -> f64 {\n    tdigest_max(sketch)\n}\n\n// Maximum value entered in the digest.\n#[pg_extern(immutable, parallel_safe, name = \"max_val\")]\npub fn tdigest_max<'a>(digest: TDigest<'a>) -> f64 {\n    digest.max\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_tdigest_mean<'a>(sketch: TDigest<'a>, _accessor: AccessorMean) -> f64 {\n    tdigest_mean(sketch)\n}\n\n// Average of all the values entered in the digest.\n// Note that this is not an approximation, though there may be loss of precision.\n#[pg_extern(immutable, parallel_safe, name = \"mean\")]\npub fn tdigest_mean<'a>(digest: TDigest<'a>) -> f64 {\n    if digest.count > 0 {\n        digest.sum / digest.count as f64\n    } else {\n        0.0\n    }\n}\n\n/// Total sum of all the values entered in the digest.\n#[pg_extern(immutable, parallel_safe, name = \"total\")]\npub fn tdigest_sum(digest: TDigest<'_>) -> f64 {\n    digest.sum\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n\n    use pgrx_macros::pg_test;\n\n    // Assert equality between two floats, within some fixed error range.\n    fn apx_eql(value: f64, expected: f64, error: f64) {\n        assert!(\n            (value - expected).abs() < error,\n            \"Float value {value} differs from expected {expected} by more than {error}\"\n        );\n    }\n\n    // Assert equality between two floats, within an error expressed as a fraction of the expected value.\n    fn pct_eql(value: f64, expected: f64, pct_error: f64) {\n        apx_eql(value, expected, pct_error * expected);\n    }\n\n    #[pg_test]\n    fn test_tdigest_aggregate() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test (data DOUBLE PRECISION)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO test SELECT generate_series(0.01, 100, 0.01)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(10000, sanity.unwrap());\n\n            client\n                .update(\n                    \"CREATE VIEW digest AS \\\n                SELECT tdigest(100, data) FROM test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (min, max, count) = client\n                .update(\n                    \"SELECT \\\n                    min_val(tdigest), \\\n                    max_val(tdigest), \\\n                    num_vals(tdigest) \\\n                    FROM digest\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_three::<f64, f64, f64>()\n                .unwrap();\n\n            apx_eql(min.unwrap(), 0.01, 0.000001);\n            apx_eql(max.unwrap(), 100.0, 0.000001);\n            apx_eql(count.unwrap(), 10000.0, 0.000001);\n\n            let (min2, max2, count2) = client\n                .update(\n                    \"SELECT \\\n                    tdigest->min_val(), \\\n                    tdigest->max_val(), \\\n                    tdigest->num_vals() \\\n                    FROM digest\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_three::<f64, f64, f64>()\n                .unwrap();\n\n            assert_eq!(min2, min);\n            assert_eq!(max2, max);\n            assert_eq!(count2, count);\n\n            let (mean, mean2) = client\n                .update(\n                    \"SELECT \\\n                    mean(tdigest), \\\n                    tdigest -> mean()\n                    FROM digest\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            apx_eql(mean.unwrap(), 50.005, 0.0001);\n            assert_eq!(mean, mean2);\n\n            for i in 0..=100 {\n                let value = i as f64;\n                let quantile = value / 100.0;\n\n                let (est_val, est_quant) = client\n                    .update(\n                        &format!(\n                            \"SELECT\n                            approx_percentile({quantile}, tdigest), \\\n                            approx_percentile_rank({value}, tdigest) \\\n                            FROM digest\"\n                        ),\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_two::<f64, f64>()\n                    .unwrap();\n\n                if i == 0 {\n                    pct_eql(est_val.unwrap(), 0.01, 1.0);\n                    apx_eql(est_quant.unwrap(), quantile, 0.0001);\n                } else {\n                    pct_eql(est_val.unwrap(), value, 1.0);\n                    pct_eql(est_quant.unwrap(), quantile, 1.0);\n                }\n\n                let (est_val2, est_quant2) = client\n                    .update(\n                        &format!(\n                            \"SELECT\n                            tdigest->approx_percentile({quantile}), \\\n                            tdigest->approx_percentile_rank({value}) \\\n                            FROM digest\"\n                        ),\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_two::<f64, f64>()\n                    .unwrap();\n                assert_eq!(est_val2, est_val);\n                assert_eq!(est_quant2, est_quant);\n            }\n        });\n    }\n\n    #[pg_test]\n    fn test_tdigest_small_count() {\n        Spi::connect_mut(|client| {\n            let estimate = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile(\\\n                        0.99, \\\n                        tdigest(100, data)) \\\n                    FROM generate_series(1, 100) data;\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one()\n                .unwrap();\n\n            assert_eq!(estimate, Some(99.5));\n        });\n    }\n\n    #[pg_test]\n    fn serialization_matches() {\n        let mut t = InternalTDigest::new_with_size(10);\n        let vals = vec![1.0, 1.0, 1.0, 2.0, 1.0, 1.0];\n        for v in vals {\n            t = t.merge_unsorted(vec![v]);\n        }\n        let pgt = TDigest::from_internal_tdigest(&t);\n        let mut si = StringInfo::new();\n        pgt.output(&mut si);\n        assert_eq!(t.format_for_postgres(), si.to_string());\n    }\n\n    #[pg_test]\n    fn test_tdigest_io() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\n                    \"SELECT \\\n                tdigest(100, data)::text \\\n                FROM generate_series(1, 100) data;\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(version:1,buckets:88,max_buckets:100,count:100,sum:5050,min:1,max:100,centroids:[(mean:1,weight:1),(mean:2,weight:1),(mean:3,weight:1),(mean:4,weight:1),(mean:5,weight:1),(mean:6,weight:1),(mean:7,weight:1),(mean:8,weight:1),(mean:9,weight:1),(mean:10,weight:1),(mean:11,weight:1),(mean:12,weight:1),(mean:13,weight:1),(mean:14,weight:1),(mean:15,weight:1),(mean:16,weight:1),(mean:17,weight:1),(mean:18,weight:1),(mean:19,weight:1),(mean:20,weight:1),(mean:21,weight:1),(mean:22,weight:1),(mean:23,weight:1),(mean:24,weight:1),(mean:25,weight:1),(mean:26,weight:1),(mean:27,weight:1),(mean:28,weight:1),(mean:29,weight:1),(mean:30,weight:1),(mean:31,weight:1),(mean:32,weight:1),(mean:33,weight:1),(mean:34,weight:1),(mean:35,weight:1),(mean:36,weight:1),(mean:37,weight:1),(mean:38,weight:1),(mean:39,weight:1),(mean:40,weight:1),(mean:41,weight:1),(mean:42,weight:1),(mean:43,weight:1),(mean:44,weight:1),(mean:45,weight:1),(mean:46,weight:1),(mean:47,weight:1),(mean:48,weight:1),(mean:49,weight:1),(mean:50,weight:1),(mean:51,weight:1),(mean:52.5,weight:2),(mean:54.5,weight:2),(mean:56.5,weight:2),(mean:58.5,weight:2),(mean:60.5,weight:2),(mean:62.5,weight:2),(mean:64,weight:1),(mean:65.5,weight:2),(mean:67.5,weight:2),(mean:69,weight:1),(mean:70.5,weight:2),(mean:72,weight:1),(mean:73.5,weight:2),(mean:75,weight:1),(mean:76,weight:1),(mean:77.5,weight:2),(mean:79,weight:1),(mean:80,weight:1),(mean:81.5,weight:2),(mean:83,weight:1),(mean:84,weight:1),(mean:85,weight:1),(mean:86,weight:1),(mean:87,weight:1),(mean:88,weight:1),(mean:89,weight:1),(mean:90,weight:1),(mean:91,weight:1),(mean:92,weight:1),(mean:93,weight:1),(mean:94,weight:1),(mean:95,weight:1),(mean:96,weight:1),(mean:97,weight:1),(mean:98,weight:1),(mean:99,weight:1),(mean:100,weight:1)])\";\n\n            assert_eq!(output, Some(expected.into()));\n\n            let estimate = client\n                .update(\n                    &format!(\"SELECT approx_percentile(0.90, '{expected}'::tdigest)\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one()\n                .unwrap();\n            assert_eq!(estimate, Some(90.5));\n        });\n    }\n\n    #[pg_test]\n    fn test_tdigest_byte_io() {\n        unsafe {\n            use std::ptr;\n            let state = tdigest_trans_inner(None, 100, Some(14.0), ptr::null_mut());\n            let state = tdigest_trans_inner(state, 100, Some(18.0), ptr::null_mut());\n            let state = tdigest_trans_inner(state, 100, Some(22.7), ptr::null_mut());\n            let state = tdigest_trans_inner(state, 100, Some(39.42), ptr::null_mut());\n            let state = tdigest_trans_inner(state, 100, Some(-43.0), ptr::null_mut());\n\n            let mut control = state.unwrap();\n            let buffer = tdigest_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = pgrx::varlena::varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let expected = [\n                1, 1, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 69, 192, 1, 0, 0, 0, 0, 0, 0, 0,\n                0, 0, 0, 0, 0, 0, 44, 64, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 50, 64, 1, 0,\n                0, 0, 0, 0, 0, 0, 51, 51, 51, 51, 51, 179, 54, 64, 1, 0, 0, 0, 0, 0, 0, 0, 246, 40,\n                92, 143, 194, 181, 67, 64, 1, 0, 0, 0, 0, 0, 0, 0, 100, 0, 0, 0, 0, 0, 0, 0, 144,\n                194, 245, 40, 92, 143, 73, 64, 5, 0, 0, 0, 0, 0, 0, 0, 246, 40, 92, 143, 194, 181,\n                67, 64, 0, 0, 0, 0, 0, 128, 69, 192,\n            ];\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let mut new_state =\n                tdigest_deserialize_inner(bytea(pg_sys::Datum::from(expected.as_ptr())));\n\n            assert_eq!(new_state.build(), control.build());\n        }\n    }\n\n    #[pg_test]\n    fn test_tdigest_compound_agg() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE new_test (device INTEGER, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"INSERT INTO new_test SELECT dev, dev - v FROM generate_series(1,10) dev, generate_series(0, 1.0, 0.01) v\", None, &[]).unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM new_test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(1010), sanity);\n\n            client\n                .update(\n                    \"CREATE VIEW digests AS \\\n                SELECT device, tdigest(20, value) \\\n                FROM new_test \\\n                GROUP BY device\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW composite AS \\\n                SELECT tdigest(tdigest) \\\n                FROM digests\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW base AS \\\n                SELECT tdigest(20, value) \\\n                FROM new_test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let value = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile(0.9, tdigest) \\\n                    FROM base\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap();\n\n            let test_value = client\n                .update(\n                    \"SELECT \\\n                approx_percentile(0.9, tdigest) \\\n                    FROM composite\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap();\n\n            apx_eql(test_value.unwrap(), value.unwrap(), 0.1);\n            apx_eql(test_value.unwrap(), 9.0, 0.1);\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/iter.rs",
    "content": "use tspoint::TSPoint;\n\nuse Iter::*;\n\npub enum Iter<'a> {\n    Slice {\n        iter: flat_serialize::Iter<'a, 'a, TSPoint>,\n    },\n}\n\nimpl<'a> Iterator for Iter<'a> {\n    type Item = TSPoint;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        match self {\n            Slice { iter } => iter.next(),\n        }\n    }\n\n    // XXX the functions below, `last()` and `count()` in particular rely on\n    //     this being precise and accurate, with both elements of the tuple\n    //     being the same as the actual yielded number of elements, if this\n    //     changes those will also need to change\n    fn size_hint(&self) -> (usize, Option<usize>) {\n        match self {\n            Slice { iter } => (iter.len(), Some(iter.len())),\n        }\n    }\n\n    fn count(self) -> usize\n    where\n        Self: Sized,\n    {\n        self.size_hint().0\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/aggregation.rs",
    "content": "use std::mem::take;\n\nuse pgrx::*;\n\nuse counter_agg::CounterSummaryBuilder;\n\nuse super::*;\n\nuse crate::{\n    accessors::{AccessorAverage, AccessorNumVals, AccessorSum},\n    build,\n    counter_agg::CounterSummary,\n    hyperloglog::HyperLogLog,\n    pg_type, ron_inout_funcs,\n    stats_agg::{self, InternalStatsSummary1D, StatsSummary1D},\n    uddsketch::UddSketch,\n};\n\nuse self::toolkit_experimental::{\n    PipelineThenAverage, PipelineThenAverageData, PipelineThenCounterAgg,\n    PipelineThenCounterAggData, PipelineThenHyperLogLog, PipelineThenHyperLogLogData,\n    PipelineThenNumVals, PipelineThenNumValsData, PipelineThenPercentileAgg,\n    PipelineThenPercentileAggData, PipelineThenStatsAgg, PipelineThenStatsAggData, PipelineThenSum,\n    PipelineThenSumData,\n};\n\n#[pg_schema]\npub mod toolkit_experimental {\n    use super::*;\n    pub(crate) use crate::time_vector::pipeline::UnstableTimevectorPipeline;\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenStatsAgg<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenStatsAgg<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenSum<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenSum<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenAverage<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenAverage<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenNumVals<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenNumVals<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenCounterAgg<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenCounterAgg<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenHyperLogLog<'input> {\n            hll_size: u64,\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenHyperLogLog<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenPercentileAgg<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenPercentileAgg<'input>);\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline_then_stats_agg<'a>(\n    mut timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenStatsAgg<'a>,\n) -> StatsSummary1D {\n    if timevector.has_nulls() {\n        panic!(\"Unable to compute stats aggregate over timevector containing nulls\");\n    }\n    timevector = run_pipeline_elements(timevector, pipeline.elements.iter());\n    let mut stats = InternalStatsSummary1D::new();\n    for TSPoint { val, .. } in timevector.iter() {\n        stats.accum(val).expect(\"error while running stats_agg\");\n    }\n    StatsSummary1D::from_internal(stats)\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn finalize_with_stats_agg<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_stats_agg: toolkit_experimental::PipelineThenStatsAgg<'e>,\n) -> toolkit_experimental::PipelineThenStatsAgg<'e> {\n    if then_stats_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenStatsAgg {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_stats_agg.elements.iter());\n    build! {\n        PipelineThenStatsAgg {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"stats_agg\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_stats_agg() -> toolkit_experimental::PipelineThenStatsAgg<'static> {\n    build! {\n        PipelineThenStatsAgg {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_stats_agg_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element =\n            PipelineThenStatsAgg::from_polymorphic_datum(new_element, false, pg_sys::Oid::INVALID)\n                .unwrap();\n        finalize_with_stats_agg(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\n// using this instead of pg_operator since the latter doesn't support schemas yet\n// FIXME there is no CREATE OR REPLACE OPERATOR need to update post-install.rs\n//       need to ensure this works with out unstable warning\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_run_pipeline_then_stats_agg\" SUPPORT toolkit_experimental.pipeline_stats_agg_support;\n\"#,\n    name = \"pipeline_stats_agg_support\",\n    requires = [pipeline_stats_agg_support],\n);\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"sum_cast\",\n    schema = \"toolkit_experimental\"\n)]\npub fn sum_pipeline_element<'a>(\n    accessor: AccessorSum,\n) -> toolkit_experimental::PipelineThenSum<'static> {\n    let _ = accessor;\n    build! {\n        PipelineThenSum {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\nextension_sql!(\n    r#\"\n    CREATE CAST (AccessorSum AS toolkit_experimental.PipelineThenSum)\n        WITH FUNCTION toolkit_experimental.sum_cast\n        AS IMPLICIT;\n\"#,\n    name = \"sum_pipe_cast\",\n    requires = [AccessorSum, PipelineThenSum, sum_pipeline_element],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_pipeline_then_sum<'a>(\n    timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenSum<'a>,\n) -> Option<f64> {\n    let pipeline = pipeline.0;\n    let pipeline = build! {\n        PipelineThenStatsAgg {\n            num_elements: pipeline.num_elements,\n            elements: pipeline.elements,\n        }\n    };\n    let stats_agg = arrow_run_pipeline_then_stats_agg(timevector, pipeline);\n    stats_agg::stats1d_sum(stats_agg)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn finalize_with_sum<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_stats_agg: toolkit_experimental::PipelineThenSum<'e>,\n) -> toolkit_experimental::PipelineThenSum<'e> {\n    if then_stats_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenSum {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_stats_agg.elements.iter());\n    build! {\n        PipelineThenSum {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_sum_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element =\n            PipelineThenSum::from_polymorphic_datum(new_element, false, pg_sys::Oid::INVALID)\n                .unwrap();\n        finalize_with_sum(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_pipeline_then_sum\" SUPPORT toolkit_experimental.pipeline_sum_support;\n\"#,\n    name = \"arrow_then_sum_support\",\n    requires = [pipeline_sum_support],\n);\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn average_pipeline_element(\n    accessor: AccessorAverage,\n) -> toolkit_experimental::PipelineThenAverage<'static> {\n    let _ = accessor;\n    build! {\n        PipelineThenAverage {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\nextension_sql!(\n    r#\"\n    CREATE CAST (AccessorAverage AS toolkit_experimental.PipelineThenAverage)\n        WITH FUNCTION toolkit_experimental.average_pipeline_element\n        AS IMPLICIT;\n\"#,\n    name = \"avg_pipe_cast\",\n    requires = [\n        AccessorAverage,\n        PipelineThenAverage,\n        average_pipeline_element\n    ],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_pipeline_then_average<'a>(\n    timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenAverage<'a>,\n) -> Option<f64> {\n    let pipeline = pipeline.0;\n    let pipeline = build! {\n        PipelineThenStatsAgg {\n            num_elements: pipeline.num_elements,\n            elements: pipeline.elements,\n        }\n    };\n    let stats_agg = arrow_run_pipeline_then_stats_agg(timevector, pipeline);\n    stats_agg::stats1d_average(stats_agg)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn finalize_with_average<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_stats_agg: toolkit_experimental::PipelineThenAverage<'e>,\n) -> toolkit_experimental::PipelineThenAverage<'e> {\n    if then_stats_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenAverage {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_stats_agg.elements.iter());\n    build! {\n        PipelineThenAverage {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_average_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element =\n            PipelineThenAverage::from_polymorphic_datum(new_element, false, pg_sys::Oid::INVALID)\n                .unwrap();\n        finalize_with_average(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_pipeline_then_average\" SUPPORT toolkit_experimental.pipeline_average_support;\n\"#,\n    name = \"pipe_avg_support\",\n    requires = [pipeline_average_support],\n);\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"num_vals_cast\",\n    schema = \"toolkit_experimental\"\n)]\npub fn num_vals_pipeline_element(\n    accessor: AccessorNumVals,\n) -> toolkit_experimental::PipelineThenNumVals<'static> {\n    let _ = accessor;\n    build! {\n        PipelineThenNumVals {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\nextension_sql!(\n    r#\"\n    CREATE CAST (AccessorNumVals AS toolkit_experimental.PipelineThenNumVals)\n        WITH FUNCTION toolkit_experimental.num_vals_cast\n        AS IMPLICIT;\n\"#,\n    name = \"num_vals_pipe_cast\",\n    requires = [\n        AccessorNumVals,\n        PipelineThenNumVals,\n        num_vals_pipeline_element\n    ],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_pipeline_then_num_vals<'a>(\n    timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenNumVals<'a>,\n) -> i64 {\n    run_pipeline_elements(timevector, pipeline.elements.iter()).num_vals() as _\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn finalize_with_num_vals<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_stats_agg: toolkit_experimental::PipelineThenNumVals<'e>,\n) -> toolkit_experimental::PipelineThenNumVals<'e> {\n    if then_stats_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenNumVals {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_stats_agg.elements.iter());\n    build! {\n        PipelineThenNumVals {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_num_vals_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element =\n            PipelineThenNumVals::from_polymorphic_datum(new_element, false, pg_sys::Oid::INVALID)\n                .unwrap();\n        finalize_with_num_vals(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_pipeline_then_num_vals\" SUPPORT toolkit_experimental.pipeline_num_vals_support;\n\"#,\n    name = \"pipe_then_num_vals\",\n    requires = [pipeline_num_vals_support],\n);\n\n// TODO support gauge\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline_then_counter_agg<'a>(\n    mut timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenCounterAgg<'a>,\n) -> Option<CounterSummary> {\n    timevector = run_pipeline_elements(timevector, pipeline.elements.iter());\n    if timevector.num_points() == 0 {\n        return None;\n    }\n    let mut it = timevector.iter();\n    let mut summary = CounterSummaryBuilder::new(&it.next().unwrap(), None);\n    for point in it {\n        summary\n            .add_point(&point)\n            .expect(\"error while running counter_agg\");\n    }\n    Some(CounterSummary::from_internal_counter_summary(\n        summary.build(),\n    ))\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn finalize_with_counter_agg<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_counter_agg: toolkit_experimental::PipelineThenCounterAgg<'e>,\n) -> toolkit_experimental::PipelineThenCounterAgg<'e> {\n    if then_counter_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenCounterAgg {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_counter_agg.elements.iter());\n    build! {\n        PipelineThenCounterAgg {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"counter_agg\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_counter_agg() -> toolkit_experimental::PipelineThenCounterAgg<'static> {\n    build! {\n        PipelineThenCounterAgg {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_counter_agg_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element = PipelineThenCounterAgg::from_polymorphic_datum(\n            new_element,\n            false,\n            pg_sys::Oid::INVALID,\n        )\n        .unwrap();\n        finalize_with_counter_agg(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\n// using this instead of pg_operator since the latter doesn't support schemas yet\n// FIXME there is no CREATE OR REPLACE OPERATOR need to update post-install.rs\n//       need to ensure this works with out unstable warning\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_run_pipeline_then_counter_agg\" SUPPORT toolkit_experimental.pipeline_counter_agg_support;\n\"#,\n    name = \"pipe_then_counter_agg\",\n    requires = [pipeline_counter_agg_support],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline_then_hyperloglog<'a>(\n    mut timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenHyperLogLog<'a>,\n) -> HyperLogLog<'static> {\n    timevector = run_pipeline_elements(timevector, pipeline.elements.iter());\n    HyperLogLog::build_from(\n        pipeline.hll_size as i32,\n        PgBuiltInOids::FLOAT8OID.into(),\n        None,\n        timevector\n            .iter()\n            .map(|point| point.val.into_datum().unwrap()),\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn finalize_with_hyperloglog<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_hyperloglog: toolkit_experimental::PipelineThenHyperLogLog<'e>,\n) -> toolkit_experimental::PipelineThenHyperLogLog<'e> {\n    if then_hyperloglog.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenHyperLogLog {\n                    hll_size: then_hyperloglog.hll_size,\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_hyperloglog.elements.iter());\n    build! {\n        PipelineThenHyperLogLog {\n            hll_size: then_hyperloglog.hll_size,\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"hyperloglog\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_hyperloglog(size: i32) -> toolkit_experimental::PipelineThenHyperLogLog<'static> {\n    build! {\n        PipelineThenHyperLogLog {\n            hll_size: size as u64,\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_hyperloglog_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element = PipelineThenHyperLogLog::from_polymorphic_datum(\n            new_element,\n            false,\n            pg_sys::Oid::INVALID,\n        )\n        .unwrap();\n        finalize_with_hyperloglog(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\n// using this instead of pg_operator since the latter doesn't support schemas yet\n// FIXME there is no CREATE OR REPLACE OPERATOR need to update post-install.rs\n//       need to ensure this works with out unstable warning\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_run_pipeline_then_hyperloglog\" SUPPORT toolkit_experimental.pipeline_hyperloglog_support;\n\"#,\n    name = \"pipe_then_hll\",\n    requires = [pipeline_hyperloglog_support],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline_then_percentile_agg<'a>(\n    mut timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenPercentileAgg<'a>,\n) -> UddSketch<'static> {\n    timevector = run_pipeline_elements(timevector, pipeline.elements.iter());\n    UddSketch::from_iter(timevector.into_iter().map(|p| p.val))\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn finalize_with_percentile_agg<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_hyperloglog: toolkit_experimental::PipelineThenPercentileAgg<'e>,\n) -> toolkit_experimental::PipelineThenPercentileAgg<'e> {\n    if then_hyperloglog.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenPercentileAgg {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_hyperloglog.elements.iter());\n    build! {\n        PipelineThenPercentileAgg {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"percentile_agg\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_percentile_agg() -> toolkit_experimental::PipelineThenPercentileAgg<'static> {\n    build! {\n        PipelineThenPercentileAgg {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_percentile_agg_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element = PipelineThenPercentileAgg::from_polymorphic_datum(\n            new_element,\n            false,\n            pg_sys::Oid::INVALID,\n        )\n        .unwrap();\n        finalize_with_percentile_agg(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\n// using this instead of pg_operator since the latter doesn't support schemas yet\n// FIXME there is no CREATE OR REPLACE OPERATOR need to update post-install.rs\n//       need to ensure this works with out unstable warning\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_run_pipeline_then_percentile_agg\" SUPPORT toolkit_experimental.pipeline_percentile_agg_support;\n\"#,\n    name = \"pipe_then_percentile\",\n    requires = [pipeline_percentile_agg_support],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_stats_agg_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> stats_agg())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,n:5,sx:100,sx2:250,sx3:0,sx4:21250)\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_stats_agg_pipeline_folding() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> stats_agg() -> average();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: (\\\n                arrow_run_pipeline_then_stats_agg(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethenstatsagg\\\n                ) -> '(version:1)'::accessoraverage)\");\n        });\n    }\n\n    #[pg_test]\n    fn test_sum_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> sum())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), \"100\");\n        });\n    }\n\n    #[pg_test]\n    fn test_sum_pipeline_folding() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> sum();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_pipeline_then_sum(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethensum\\\n                )\");\n        });\n    }\n\n    #[pg_test]\n    fn test_average_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> average())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), \"20\");\n        });\n    }\n\n    #[pg_test]\n    fn test_average_pipeline_folding() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> average();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_pipeline_then_average(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethenaverage\\\n                )\");\n        });\n    }\n\n    #[pg_test]\n    fn test_num_vals_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> num_vals())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), \"5\");\n        });\n    }\n\n    #[pg_test]\n    fn test_num_vals_pipeline_folding() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> num_vals();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_pipeline_then_num_vals(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethennumvals\\\n                )\");\n        });\n    }\n\n    #[pg_test]\n    fn test_counter_agg_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n            (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 10.0), \\\n                ('2020-01-01 UTC'::TIMESTAMPTZ, 15.0), \\\n                ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                ('2020-01-02 UTC'::TIMESTAMPTZ, 25.0), \\\n                ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\n                        \"SELECT (series -> sort() -> counter_agg())::TEXT FROM ({create_series}) s\"\n                    ),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), \"(version:1,stats:(n:5,sx:3156624000,sx2:74649600000,sx3:0,sx4:1894671345254400000000,sy:215,sy2:2280,sy3:6720.000000000007,sy4:1788960,sxy:12960000),first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:15),second:(ts:\\\"2020-01-02 00:00:00+00\\\",val:25),penultimate:(ts:\\\"2020-01-04 00:00:00+00\\\",val:10),last:(ts:\\\"2020-01-05 00:00:00+00\\\",val:30),reset_sum:45,num_resets:2,num_changes:4,bounds:(is_present:0,has_left:0,has_right:0,padding:(0,0,0,0,0),left:None,right:None))\");\n\n            let val = client.update(\n                &format!(\"SELECT series -> sort() -> counter_agg() -> with_bounds('[2020-01-01 UTC, 2020-02-01 UTC)') -> extrapolated_delta('prometheus') FROM ({create_series}) s\"),\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<f64>().unwrap().unwrap();\n            assert!((val - 67.5).abs() < f64::EPSILON);\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> counter_agg();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_run_pipeline_then_counter_agg(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethencounteragg\\\n                )\");\n        })\n    }\n\n    #[pg_test]\n    fn test_hyperloglog_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n            (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 10.0), \\\n                ('2020-01-01 UTC'::TIMESTAMPTZ, 15.0), \\\n                ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                ('2020-01-02 UTC'::TIMESTAMPTZ, 25.0), \\\n                ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0), \\\n                ('2020-01-06 UTC'::TIMESTAMPTZ, 25.0), \\\n                ('2020-01-07 UTC'::TIMESTAMPTZ, 15.0), \\\n                ('2020-01-08 UTC'::TIMESTAMPTZ, 35.0), \\\n                ('2020-01-09 UTC'::TIMESTAMPTZ, 10.0), \\\n                ('2020-01-10 UTC'::TIMESTAMPTZ, 5.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> hyperloglog(100))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), \"(version:1,log:Sparse(num_compressed:7,element_type:FLOAT8,collation:None,compressed_bytes:28,precision:7,compressed:[136,188,20,7,8,30,244,43,72,69,89,2,72,255,97,27,72,83,248,27,200,110,35,5,8,37,85,12]))\");\n\n            let val = client\n                .update(\n                    &format!(\n                        \"SELECT series -> hyperloglog(100) -> distinct_count() FROM ({create_series}) s\"\n                    ),\n                    None,\n                    &[]\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(val, 7);\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> hyperloglog(100);\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_run_pipeline_then_hyperloglog(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,hll_size:100,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethenhyperloglog\\\n                )\");\n        })\n    }\n\n    #[pg_test]\n    fn test_percentile_agg_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> percentile_agg())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,\\\n                    alpha:0.001,\\\n                    max_buckets:200,\\\n                    num_buckets:5,\\\n                    compactions:0,\\\n                    count:5,\\\n                    sum:100,\\\n                    buckets:[\\\n                        (Positive(1152),1),\\\n                        (Positive(1355),1),\\\n                        (Positive(1498),1),\\\n                        (Positive(1610),1),\\\n                        (Positive(1701),1)\\\n                    ]\\\n                )\",\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_percentile_agg_pipeline_folding() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('1930-04-05'::timestamptz, 123.0) \\\n                -> ceil() -> abs() -> floor() \\\n                -> percentile_agg();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_run_pipeline_then_percentile_agg(\\\n                    timevector('1930-04-05 00:00:00+00'::timestamp with time zone, '123'::double precision), \\\n                    '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Ceil,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Floor,rhs:0)\\\n                    ])'::pipelinethenpercentileagg\\\n                )\");\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/arithmetic.rs",
    "content": "use pgrx::*;\n\nuse super::*;\n\nuse super::Element::Arithmetic;\nuse Function::*;\n\n#[derive(\n    Debug, Copy, Clone, flat_serialize_macro::FlatSerializable, serde::Serialize, serde::Deserialize,\n)]\n#[repr(u64)]\n//XXX note that the order here _is_ significant; it can be visible in the\n//    serialized form\npub enum Function {\n    // binary functions\n    Add = 1,\n    Sub = 2,\n    Mul = 3,\n    Div = 4,\n    Mod = 5,\n    Power = 6,\n    LogN = 7,\n    // Unary functions\n    Abs,\n    Cbrt,\n    Ceil,\n    Floor,\n    Ln,\n    Log10,\n    Round, // nearest\n    Sign,\n    Sqrt,\n    Trunc,\n}\n\npub fn apply(\n    mut series: Timevector_TSTZ_F64<'_>,\n    function: Function,\n    rhs: f64,\n) -> Timevector_TSTZ_F64<'_> {\n    let function: fn(f64, f64) -> f64 = match function {\n        Add => |a, b| a + b,\n        Sub => |a, b| a - b,\n        Mul => |a, b| a * b,\n        Div => |a, b| a / b,\n        // TODO is this the right mod?\n        Mod => |a, b| a % b,\n        Power => |a, b| a.powf(b),\n        LogN => |a, b| a.log(b),\n        // unary functions just ignore the second arg\n        Abs => |a, _| a.abs(),\n        Cbrt => |a, _| a.cbrt(),\n        Ceil => |a, _| a.ceil(),\n        Floor => |a, _| a.floor(),\n        Ln => |a, _| a.ln(),\n        Log10 => |a, _| a.log10(),\n        Round => |a, _| a.round(),\n        Sign => |a, _| a.signum(),\n        Sqrt => |a, _| a.sqrt(),\n        Trunc => |a, _| a.trunc(),\n    };\n    map::map_series(&mut series, |lhs| function(lhs, rhs));\n    series\n}\n\n//\n// binary operations\n//\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"add\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_add(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic { function: Add, rhs }.flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"sub\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_sub(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic { function: Sub, rhs }.flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"mul\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_mul(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic { function: Mul, rhs }.flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"div\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_div(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic { function: Div, rhs }.flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"mod\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_mod(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic { function: Mod, rhs }.flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"power\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_power(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Power,\n        rhs,\n    }\n    .flatten()\n}\n\n// log(double) already exists as the log base 10 so we need a new name\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"logn\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_log_n(rhs: f64) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: LogN,\n        rhs,\n    }\n    .flatten()\n}\n\n//\n// unary operations\n//\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"abs\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_abs() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Abs,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"cbrt\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_cbrt() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Cbrt,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"ceil\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_ceil() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Ceil,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"floor\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_floor() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Floor,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"ln\", schema = \"toolkit_experimental\")]\npub fn pipeline_ln() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Ln,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"log10\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_log10() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Log10,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"round\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_round() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Round,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"sign\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_sign() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Sign,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"sqrt\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_sqrt() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Sqrt,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"trunc\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_trunc() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Arithmetic {\n        function: Trunc,\n        rhs: 0.0,\n    }\n    .flatten()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_simple_arith_binops() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> add(1.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:26),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:11),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:21),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:16),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:31)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> sub(3.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:22),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:7),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:17),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:12),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:27)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> mul(2.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:50),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:60)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> div(5.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:5),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:2),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:4),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:3),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:6)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> mod(5.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:0),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:0),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:0),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:0),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:0)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> power(2.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:625),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:100),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:400),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:225),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:900)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> logn(10.0))::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:1.3979400086720375),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:1),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:1.301029995663981),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:1.1760912590556811),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:1.4771212547196624)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_simple_arith_unaryops() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.5), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, -10.1), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.2), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, -15.6), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.3)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> abs())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25.5),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10.1),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20.2),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15.6),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30.3)\\\n            ],null_val:[0])\"\n            );\n\n            // TODO re-enable once made stable\n            // let val = client.update(\n            //     &format!(\"SELECT (series -> cbrt())::TEXT FROM ({}) s\", create_series),\n            //     None,\n            //     None\n            // )\n            //     .first()\n            //     .get_one::<String>().unwrap();\n            // assert_eq!(val.unwrap(), \"[\\\n            //     (ts:\\\"2020-01-04 00:00:00+00\\\",val:2.943382658441668),\\\n            //     (ts:\\\"2020-01-01 00:00:00+00\\\",val:-2.161592332945083),\\\n            //     (ts:\\\"2020-01-03 00:00:00+00\\\",val:2.7234356815688767),\\\n            //     (ts:\\\"2020-01-02 00:00:00+00\\\",val:-2.4986659549227817),\\\n            //     (ts:\\\"2020-01-05 00:00:00+00\\\",val:3.117555613369834)\\\n            // ]\");\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> ceil())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:26),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:-10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:21),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:-15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:31)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> floor())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:-11),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:-16),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            // TODO why are there `null`s here?\n            // Josh - likely JSON can't represent nans correctly...\n            // TODO re-enable once made stable\n            // let val = client.update(\n            //     &format!(\"SELECT (series -> ln())::TEXT FROM ({}) s\", create_series),\n            //     None,\n            //     None\n            // )\n            //     .first()\n            //     .get_one::<String>().unwrap();\n            // assert_eq!(val.unwrap(), \"[\\\n            //     (ts:\\\"2020-01-04 00:00:00+00\\\",val:3.2386784521643803),\\\n            //     (ts:\\\"2020-01-01 00:00:00+00\\\",val:null),\\\n            //     (ts:\\\"2020-01-03 00:00:00+00\\\",val:3.005682604407159),\\\n            //     (ts:\\\"2020-01-02 00:00:00+00\\\",val:null),\\\n            //     (ts:\\\"2020-01-05 00:00:00+00\\\",val:3.4111477125153233)\\\n            // ]\");\n\n            // TODO re-enable once made stable\n            // let val = client.update(\n            //     &format!(\"SELECT (series -> log10())::TEXT FROM ({}) s\", create_series),\n            //     None,\n            //     None\n            // )\n            //     .first()\n            //     .get_one::<String>().unwrap();\n            // assert_eq!(val.unwrap(), \"[\\\n            //     (ts:\\\"2020-01-04 00:00:00+00\\\",val:1.4065401804339552),\\\n            //     (ts:\\\"2020-01-01 00:00:00+00\\\",val:null),\\\n            //     (ts:\\\"2020-01-03 00:00:00+00\\\",val:1.3053513694466237),\\\n            //     (ts:\\\"2020-01-02 00:00:00+00\\\",val:null),\\\n            //     (ts:\\\"2020-01-05 00:00:00+00\\\",val:1.481442628502305)\\\n            // ]\");\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> round())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:26),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:-10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:-16),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> sign())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:1),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:-1),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:1),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:-1),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:1)\\\n            ],null_val:[0])\"\n            );\n\n            // TODO re-enable once made stable\n            // let val = client.update(\n            //     &format!(\"SELECT (series -> sqrt())::TEXT FROM ({}) s\", create_series),\n            //     None,\n            //     None\n            // )\n            //     .first()\n            //     .get_one::<String>().unwrap();\n            // assert_eq!(val.unwrap(), \"[\\\n            //     (ts:\\\"2020-01-04 00:00:00+00\\\",val:5.049752469181039),\\\n            //     (ts:\\\"2020-01-01 00:00:00+00\\\",val:null),\\\n            //     (ts:\\\"2020-01-03 00:00:00+00\\\",val:4.494441010848846),\\\n            //     (ts:\\\"2020-01-02 00:00:00+00\\\",val:null),\\\n            //     (ts:\\\"2020-01-05 00:00:00+00\\\",val:5.504543577809154)\\\n            // ]\");\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> trunc())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:-10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:-15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/delta.rs",
    "content": "use pgrx::*;\n\nuse super::*;\n\nuse crate::accessors::AccessorDelta;\n\n// TODO is (immutable, parallel_safe) correct?\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"delta_cast\",\n    schema = \"toolkit_experimental\"\n)]\npub fn delta_pipeline_element(\n    accessor: AccessorDelta,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    let _ = accessor;\n    Element::Delta {}.flatten()\n}\n\nextension_sql!(\n    r#\"\n    CREATE CAST (AccessorDelta AS toolkit_experimental.UnstableTimevectorPipeline)\n        WITH FUNCTION toolkit_experimental.delta_cast\n        AS IMPLICIT;\n\"#,\n    name = \"accessor_delta_cast\",\n    requires = [delta_pipeline_element]\n);\n\npub fn timevector_delta<'s>(series: &Timevector_TSTZ_F64<'s>) -> Timevector_TSTZ_F64<'s> {\n    if !series.is_sorted() {\n        panic!(\"can only compute deltas for sorted timevector\");\n    }\n    if series.has_nulls() {\n        panic!(\"Unable to compute deltas over timevector containing nulls\");\n    }\n\n    let mut it = series.iter();\n    let mut prev = it.next().unwrap().val;\n    let mut delta_points = Vec::new();\n\n    for pt in it {\n        delta_points.push(TSPoint {\n            ts: pt.ts,\n            val: pt.val - prev,\n        });\n        prev = pt.val;\n    }\n\n    let nulls_len = delta_points.len().div_ceil(8);\n\n    build!(Timevector_TSTZ_F64 {\n        num_points: delta_points.len() as u32,\n        flags: series.flags,\n        internal_padding: [0; 3],\n        points: delta_points.into(),\n        null_val: std::vec::from_elem(0_u8, nulls_len).into(),\n    })\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_pipeline_delta() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 92.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.8), \\\n                    ('2020-01-06 UTC'::TIMESTAMPTZ, 30.8), \\\n                    ('2020-01-07 UTC'::TIMESTAMPTZ, 30.8), \\\n                    ('2020-01-08 UTC'::TIMESTAMPTZ, 30.9), \\\n                    ('2020-01-09 UTC'::TIMESTAMPTZ, -427.2)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value) -> delta())::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:8,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:-5),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:72),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:-61.2),\\\n                (ts:\\\"2020-01-06 00:00:00+00\\\",val:0),\\\n                (ts:\\\"2020-01-07 00:00:00+00\\\",val:0),\\\n                (ts:\\\"2020-01-08 00:00:00+00\\\",val:0.09999999999999787),\\\n                (ts:\\\"2020-01-09 00:00:00+00\\\",val:-458.09999999999997)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/expansion.rs",
    "content": "use std::mem::take;\n\nuse pgrx::{iter::TableIterator, *};\n\nuse super::*;\n\nuse crate::{build, pg_type, ron_inout_funcs};\n\nuse self::toolkit_experimental::{\n    PipelineForceMaterialize, PipelineForceMaterializeData, PipelineThenUnnest,\n    PipelineThenUnnestData,\n};\n\n#[pg_schema]\npub mod toolkit_experimental {\n    pub(crate) use super::*;\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineThenUnnest<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineThenUnnest<'input>);\n\n    pg_type! {\n        #[derive(Debug)]\n        struct PipelineForceMaterialize<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    ron_inout_funcs!(PipelineForceMaterialize<'input>);\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"unnest\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_unnest() -> toolkit_experimental::PipelineThenUnnest<'static> {\n    build! {\n        PipelineThenUnnest {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_finalize_with_unnest<'p>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'p>,\n    then_stats_agg: toolkit_experimental::PipelineThenUnnest<'p>,\n) -> toolkit_experimental::PipelineThenUnnest<'p> {\n    if then_stats_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineThenUnnest {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_stats_agg.elements.iter());\n    build! {\n        PipelineThenUnnest {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline_then_unnest<'a>(\n    timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineThenUnnest<'a>,\n) -> TableIterator<'static, (name!(time, crate::raw::TimestampTz), name!(value, f64))> {\n    let series = run_pipeline_elements(timevector, pipeline.elements.iter())\n        .0\n        .into_owned();\n    crate::time_vector::unnest(series.into())\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"materialize\",\n    schema = \"toolkit_experimental\"\n)]\npub fn pipeline_series() -> toolkit_experimental::PipelineForceMaterialize<'static> {\n    build! {\n        PipelineForceMaterialize {\n            num_elements: 0,\n            elements: vec![].into(),\n        }\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_force_materialize<'e>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'e>,\n    then_stats_agg: toolkit_experimental::PipelineForceMaterialize<'e>,\n) -> toolkit_experimental::PipelineForceMaterialize<'e> {\n    if then_stats_agg.num_elements == 0 {\n        // flatten immediately so we don't need a temporary allocation for elements\n        return unsafe {\n            flatten! {\n                PipelineForceMaterialize {\n                    num_elements: pipeline.0.num_elements,\n                    elements: pipeline.0.elements,\n                }\n            }\n        };\n    }\n\n    let mut elements = take(pipeline.elements.as_owned());\n    elements.extend(then_stats_agg.elements.iter());\n    build! {\n        PipelineForceMaterialize {\n            num_elements: elements.len().try_into().unwrap(),\n            elements: elements.into(),\n        }\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline_then_materialize<'a>(\n    timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::PipelineForceMaterialize<'a>,\n) -> toolkit_experimental::Timevector_TSTZ_F64<'static> {\n    run_pipeline_elements(timevector, pipeline.elements.iter()).in_current_context()\n}\n\n#[pg_extern(immutable, parallel_safe, schema = \"toolkit_experimental\")]\npub unsafe fn pipeline_materialize_support(input: pgrx::Internal) -> pgrx::Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element = PipelineForceMaterialize::from_polymorphic_datum(\n            new_element,\n            false,\n            pg_sys::Oid::INVALID,\n        )\n        .unwrap();\n        arrow_force_materialize(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_run_pipeline_then_materialize\" SUPPORT toolkit_experimental.pipeline_materialize_support;\n\"#,\n    name = \"pipe_then_materialize\",\n    requires = [pipeline_materialize_support],\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_unnest_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\n                        \"SELECT array_agg(val)::TEXT \\\n                    FROM (SELECT series -> unnest() as val FROM ({create_series}) s) t\"\n                    ),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), \"{\\\"(\\\\\\\"2020-01-04 00:00:00+00\\\\\\\",25)\\\",\\\"(\\\\\\\"2020-01-01 00:00:00+00\\\\\\\",10)\\\",\\\"(\\\\\\\"2020-01-03 00:00:00+00\\\\\\\",20)\\\",\\\"(\\\\\\\"2020-01-02 00:00:00+00\\\\\\\",15)\\\",\\\"(\\\\\\\"2020-01-05 00:00:00+00\\\\\\\",30)\\\"}\");\n        });\n    }\n\n    #[pg_test]\n    fn test_series_finalizer() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // we use a subselect to guarantee order\n            let create_series = \"SELECT timevector(time, value) as series FROM \\\n                (VALUES ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 11.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 21.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 31.0)) as v(time, value)\";\n\n            let val = client\n                .update(\n                    &format!(\"SELECT (series -> materialize())::TEXT FROM ({create_series}) s\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:11),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:21),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:31)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_force_materialize() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            // `-> materialize()` should force materialization, but otherwise the\n            // pipeline-folding optimization should proceed\n            let output = client\n                .update(\n                    \"EXPLAIN (verbose) SELECT \\\n                timevector('2021-01-01'::timestamptz, 0.1) \\\n                -> round() -> abs() \\\n                -> materialize() \\\n                -> abs() -> round();\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1)\n                .unwrap()\n                .value::<String>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_run_pipeline(\\\n                    arrow_run_pipeline_then_materialize(\\\n                        timevector('2021-01-01 00:00:00+00'::timestamp with time zone, '0.1'::double precision), \\\n                        '(version:1,num_elements:2,elements:[\\\n                            Arithmetic(function:Round,rhs:0),Arithmetic(function:Abs,rhs:0)\\\n                        ])'::pipelineforcematerialize\\\n                    ), \\\n                    '(version:1,num_elements:2,elements:[\\\n                        Arithmetic(function:Abs,rhs:0),Arithmetic(function:Round,rhs:0)\\\n                    ])'::unstabletimevectorpipeline\\\n                )\");\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/fill_to.rs",
    "content": "use pgrx::*;\n\nuse flat_serialize_macro::FlatSerializable;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\n\n// TODO: there are one or two other gapfill objects in this extension, these should be unified\n#[derive(Clone, Serialize, Deserialize, PartialEq, Eq, Debug, FlatSerializable)]\n#[repr(u64)]\npub enum FillToMethod {\n    Locf,\n    Interpolate,\n    Nearest,\n}\n\nimpl FillToMethod {\n    pub fn fill_point(&self, lhs: &TSPoint, rhs: &TSPoint, target_ts: i64) -> TSPoint {\n        match *self {\n            FillToMethod::Locf => TSPoint {\n                ts: target_ts,\n                val: lhs.val,\n            },\n            FillToMethod::Interpolate => {\n                let interval = rhs.ts as f64 - lhs.ts as f64;\n                let left_wt = 1. - (target_ts - lhs.ts) as f64 / interval;\n                let right_wt = 1. - (rhs.ts - target_ts) as f64 / interval;\n                TSPoint {\n                    ts: target_ts,\n                    val: lhs.val * left_wt + rhs.val * right_wt,\n                }\n            }\n            FillToMethod::Nearest => {\n                if rhs.ts - target_ts >= target_ts - lhs.ts {\n                    TSPoint {\n                        ts: target_ts,\n                        val: lhs.val,\n                    }\n                } else {\n                    TSPoint {\n                        ts: target_ts,\n                        val: rhs.val,\n                    }\n                }\n            }\n        }\n    }\n}\n\n// TODO is (immutable, parallel_safe) correct?\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"fill_to\",\n    schema = \"toolkit_experimental\"\n)]\npub fn fillto_pipeline_element(\n    interval: crate::raw::Interval,\n    fill_method: String,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    unsafe {\n        let interval = interval.0.cast_mut_ptr::<pg_sys::Interval>() as *const pg_sys::Interval;\n        // TODO: store the postgres interval object and use postgres timestamp/interval functions\n        let interval =\n            ((*interval).month as i64 * 30 + (*interval).day as i64) * 24 * 60 * 60 * 1000000\n                + (*interval).time;\n\n        let fill_method = match fill_method.to_lowercase().as_str() {\n            \"locf\" => FillToMethod::Locf,\n            \"interpolate\" => FillToMethod::Interpolate,\n            \"linear\" => FillToMethod::Interpolate,\n            \"nearest\" => FillToMethod::Nearest,\n            _ => panic!(\"Invalid fill method\"),\n        };\n\n        Element::FillTo {\n            interval,\n            fill_method,\n        }\n        .flatten()\n    }\n}\n\npub fn fill_to<'s>(\n    series: Timevector_TSTZ_F64<'s>,\n    element: &toolkit_experimental::Element,\n) -> Timevector_TSTZ_F64<'s> {\n    let (interval, method) = match element {\n        Element::FillTo {\n            interval,\n            fill_method,\n        } => (*interval, fill_method),\n        _ => unreachable!(),\n    };\n\n    if !series.is_sorted() {\n        panic!(\"Timevector must be sorted prior to passing to fill_to\")\n    }\n\n    if series.has_nulls() {\n        // TODO: This should be supportable outside of FillMode::Interpolate\n        panic!(\"Fill_to requires a timevector to not have NULL values\")\n    }\n\n    let mut result = vec![];\n    let mut it = series.iter().peekable();\n    let mut current = it.next();\n\n    while let (Some(lhs), Some(rhs)) = (current, it.peek()) {\n        if rhs.ts - lhs.ts > interval {\n            let mut target = lhs.ts + interval;\n            while target < rhs.ts {\n                result.push(method.fill_point(&lhs, rhs, target));\n                target += interval;\n            }\n        }\n\n        current = it.next();\n    }\n\n    if result.is_empty() {\n        return series;\n    }\n\n    let mut result: Vec<TSPoint> = series.iter().chain(result.into_iter()).collect();\n    result.sort_by_key(|p| p.ts);\n\n    let nulls_len = result.len().div_ceil(8);\n    build! {\n        Timevector_TSTZ_F64 {\n            num_points: result.len() as _,\n            flags: series.flags,\n            internal_padding: [0; 3],\n            points: result.into(),\n            null_val: std::vec::from_elem(0_u8, nulls_len).into(),\n        }\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_pipeline_fill_to() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 90.0), \\\n                    ('2020-01-06 UTC'::TIMESTAMPTZ, 30),   \\\n                    ('2020-01-09 UTC'::TIMESTAMPTZ, 40.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client.update(\n                \"SELECT (timevector(time, value) -> fill_to('24 hours', 'locf'))::TEXT FROM series\",\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:9,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-06 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-07 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-08 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-09 00:00:00+00\\\",val:40)\\\n            ],null_val:[0,0])\"\n            );\n\n            let val = client.update(\n                \"SELECT (timevector(time, value) -> fill_to('24 hours', 'linear'))::TEXT FROM series\",\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:9,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:60),\\\n                (ts:\\\"2020-01-06 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-07 00:00:00+00\\\",val:33.33333333333334),\\\n                (ts:\\\"2020-01-08 00:00:00+00\\\",val:36.66666666666667),\\\n                (ts:\\\"2020-01-09 00:00:00+00\\\",val:40)\\\n            ],null_val:[0,0])\"\n            );\n\n            let val = client.update(\n                \"SELECT (timevector(time, value) -> fill_to('24 hours', 'nearest'))::TEXT FROM series\",\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:9,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-06 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-07 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-08 00:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-09 00:00:00+00\\\",val:40)\\\n            ],null_val:[0,0])\"\n            );\n\n            let val = client.update(\n                \"SELECT (timevector(time, value) -> fill_to('10 hours', 'nearest'))::TEXT FROM series\",\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:22,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-01 10:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-01 20:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-02 06:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 16:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-03 10:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-03 20:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-04 10:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-04 20:00:00+00\\\",val:90),\\\n                (ts:\\\"2020-01-05 06:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-05 16:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-06 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-06 10:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-06 20:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-07 06:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-07 16:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-08 02:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-08 12:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-08 22:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-09 00:00:00+00\\\",val:40)\\\n            ],null_val:[0,0,0])\"\n            );\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/filter.rs",
    "content": "use pgrx::*;\n\nuse super::*;\n\n// TODO is (stable, parallel_safe) correct?\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"filter\",\n    schema = \"toolkit_experimental\"\n)]\npub fn filter_lambda_pipeline_element<'l>(\n    lambda: toolkit_experimental::Lambda<'l>,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    let expression = lambda.parse();\n    if expression.ty() != &lambda::Type::Bool {\n        panic!(\"invalid lambda type: the lambda must return a BOOLEAN\")\n    }\n\n    Element::FilterLambda {\n        lambda: lambda.into_data(),\n    }\n    .flatten()\n}\n\npub fn apply_lambda_to<'a>(\n    mut series: Timevector_TSTZ_F64<'a>,\n    lambda: &lambda::LambdaData<'_>,\n) -> Timevector_TSTZ_F64<'a> {\n    let expression = lambda.parse();\n    if expression.ty() != &lambda::Type::Bool {\n        panic!(\"invalid lambda type: the lambda must return a BOOLEAN\")\n    }\n\n    let mut executor = lambda::ExpressionExecutor::new(&expression);\n\n    let invoke = |time: i64, value: f64| {\n        use lambda::Value::*;\n        executor.reset();\n        let result = executor.exec(value, time);\n        match result {\n            Bool(b) => b,\n            _ => unreachable!(),\n        }\n    };\n\n    filter_lambda_over_series(&mut series, invoke);\n    series\n}\n\npub fn filter_lambda_over_series(\n    series: &mut Timevector_TSTZ_F64<'_>,\n    mut func: impl FnMut(i64, f64) -> bool,\n) {\n    series.points.as_owned().retain(|p| func(p.ts, p.val));\n    series.num_points = series.points.len() as _;\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_pipeline_filter_lambda() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client.update(\n                \"SELECT (timevector(time, value) -> filter($$ $time != '2020-01-05't and ($value = 10 or $value = 20) $$))::TEXT FROM series\",\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:2,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/lambda/executor.rs",
    "content": "use pgrx::*;\n\nuse super::*;\n\npub struct ExpressionExecutor<'e, T> {\n    exprs: &'e Expression,\n    var_vals: Vec<Option<Value>>,\n    tracer: T,\n}\n\nimpl<'e> ExpressionExecutor<'e, ()> {\n    pub fn new(exprs: &'e Expression) -> Self {\n        Self::with_tracer(exprs, ())\n    }\n}\n\nimpl<'e, T> ExpressionExecutor<'e, T>\nwhere\n    T: Tracer,\n{\n    pub fn with_fn_tracer(exprs: &'e Expression, tracer: T) -> Self\n    where\n        T: FnMut(&ExpressionSegment, &Value),\n    {\n        Self::with_tracer(exprs, tracer)\n    }\n\n    pub fn with_tracer(exprs: &'e Expression, tracer: T) -> Self {\n        Self {\n            var_vals: vec![None; exprs.variables.len()],\n            exprs,\n            tracer,\n        }\n    }\n\n    pub fn reset(&mut self) {\n        for v in &mut self.var_vals {\n            *v = None\n        }\n    }\n\n    pub fn exec(&mut self, value: f64, time: i64) -> Value {\n        self.exec_expression(&self.exprs.expr, value, time)\n    }\n\n    fn exec_expression(\n        &mut self,\n        expr: &ExpressionSegment,\n        value: f64,\n        time: i64,\n        // trace_function: impl FnMut(&ExpressionSegment, &Value),\n    ) -> Value {\n        use ExpressionSegment::*;\n        let res = match expr {\n            ValueVar => Value::Double(value),\n            TimeVar => Value::Time(time),\n            DoubleConstant(f) => Value::Double(*f),\n            TimeConstant(t) => Value::Time(*t),\n            IntervalConstant(i) => Value::Interval(*i),\n\n            UserVar(i, _) => self.force_var(*i, value, time),\n\n            FunctionCall(function, args) => self.exec_function(function, args, value, time),\n\n            Unary(op, expr, ty) => self.exec_unary_op(*op, ty, expr, value, time),\n\n            Binary(op, left, right, ty) => self.exec_binary_op(*op, ty, left, right, value, time),\n\n            BuildTuple(exprs, _) => Value::Tuple(\n                exprs\n                    .iter()\n                    .map(|e| self.exec_expression(e, value, time))\n                    .collect(),\n            ),\n        };\n        self.tracer.trace(expr, &res);\n        res\n    }\n\n    fn force_var(&mut self, i: usize, value: f64, time: i64) -> Value {\n        if let Some(value) = &self.var_vals[i] {\n            return value.clone();\n        }\n\n        let value = self.exec_expression(&self.exprs.variables[i], value, time);\n        self.var_vals[i] = Some(value.clone());\n        value\n    }\n\n    fn exec_function(\n        &mut self,\n        function: &Function,\n        args: &[ExpressionSegment],\n        value: f64,\n        time: i64,\n    ) -> Value {\n        use Function::*;\n        macro_rules! unary_function {\n            ($func:ident ( )) => {{\n                let then = self.exec_expression(&args[0], value, time).float();\n                then.$func().into()\n            }};\n        }\n        macro_rules! binary_function {\n            ($func:ident ( )) => {{\n                let args = &args[0..2];\n                let a = self.exec_expression(&args[0], value, time).float();\n                let b = self.exec_expression(&args[1], value, time).float();\n                a.$func(b).into()\n            }};\n        }\n        match function {\n            Abs => unary_function!(abs()),\n            Cbrt => unary_function!(cbrt()),\n            Ceil => unary_function!(ceil()),\n            Floor => unary_function!(floor()),\n            Ln => unary_function!(ln()),\n            Log10 => unary_function!(log10()),\n            Log => {\n                let base = self.exec_expression(&args[1], value, time).float();\n                let a = self.exec_expression(&args[0], value, time).float();\n                a.log(base).into()\n            }\n            Pi => std::f64::consts::PI.into(),\n            Round => unary_function!(round()),\n            Sign => unary_function!(signum()),\n            Sqrt => unary_function!(sqrt()),\n            Trunc => unary_function!(trunc()),\n            Acos => unary_function!(acos()),\n            Asin => unary_function!(asin()),\n            Atan => unary_function!(atan()),\n            Atan2 => binary_function!(atan2()),\n            Cos => unary_function!(cos()),\n            Sin => unary_function!(sin()),\n            Tan => unary_function!(tan()),\n            Sinh => unary_function!(sinh()),\n            Cosh => unary_function!(cosh()),\n            Tanh => unary_function!(tanh()),\n            Asinh => unary_function!(asinh()),\n            Acosh => unary_function!(acosh()),\n            Atanh => unary_function!(atanh()),\n        }\n    }\n\n    fn exec_unary_op(\n        &mut self,\n        op: UnaryOp,\n        ty: &Type,\n        expr: &ExpressionSegment,\n        value: f64,\n        time: i64,\n    ) -> Value {\n        use Type::*;\n        use UnaryOp::*;\n        match op {\n            Not => {\n                let val = self.exec_expression(expr, value, time).bool();\n                (!val).into()\n            }\n            Negative => {\n                match ty {\n                    Double => {\n                        let val = self.exec_expression(expr, value, time).float();\n                        (-val).into()\n                    }\n                    // TODO interval?\n                    _ => unreachable!(),\n                }\n            }\n        }\n    }\n\n    fn exec_binary_op(\n        &mut self,\n        op: BinOp,\n        ty: &Type,\n        left: &ExpressionSegment,\n        right: &ExpressionSegment,\n        value: f64,\n        time: i64,\n    ) -> Value {\n        use BinOp::*;\n        use Type::*;\n\n        // FIXME pgrx wraps all functions in rust wrappers, which makes them\n        //       uncallable with DirectFunctionCall(). Is there a way to\n        //       export both?\n        // TODO This is fixed in a newer pgrx version, should remove after upgrade\n\n        // XXX `NodeTag` somewhere inside `pg_sys::FunctionCallInfo` triggers\n        // `improper_ctypes` lint. The `pgrx` author explains the issue in\n        // details here:\n        //\n        // https://github.com/rust-lang/rust/issues/116831\n        //\n        // For now it seems OK to suppress these warnings here with\n        // #[allow(improper_ctypes)]\n        #[allow(improper_ctypes)]\n        unsafe extern \"C-unwind\" {\n            fn interval_pl(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n            fn interval_mi(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n            fn interval_mul(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n            fn interval_div(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n\n            fn timestamptz_pl_interval(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n            fn timestamptz_mi_interval(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n        }\n\n        macro_rules! float_op {\n            (($left: ident, $right: ident) $calc: expr) => {{\n                let $left = self.exec_expression(left, value, time).float();\n                let $right = self.exec_expression(right, value, time).float();\n                ($calc).into()\n            }};\n        }\n\n        macro_rules! interval_op {\n            (($left: ident, $right: ident) $calc: ident) => {{\n                let left = self.exec_expression(left, value, time).interval();\n                let right = self.exec_expression(right, value, time).interval();\n\n                let res: *mut pg_sys::Interval = unsafe {\n                    pg_sys::DirectFunctionCall2Coll(\n                        Some($calc),\n                        pg_sys::InvalidOid,\n                        pg_sys::Datum::from(left),\n                        pg_sys::Datum::from(right),\n                    )\n                    .cast_mut_ptr()\n                };\n                assert!(!res.is_null());\n                Value::Interval(res)\n            }};\n        }\n\n        macro_rules! interval_float_op {\n            (($left: ident, $right: ident) $calc: ident) => {{\n                let left = self.exec_expression(left, value, time).interval();\n                let right = self.exec_expression(right, value, time).float();\n\n                let res: *mut pg_sys::Interval = unsafe {\n                    pg_sys::DirectFunctionCall2Coll(\n                        Some($calc),\n                        pg_sys::InvalidOid,\n                        pg_sys::Datum::from(left),\n                        right.into_datum().unwrap(),\n                    )\n                    .value() as _\n                };\n                assert!(!res.is_null());\n                Value::Interval(res)\n            }};\n        }\n\n        macro_rules! time_op {\n            (($left: ident, $right: ident) $calc: ident) => {{\n                let left = self.exec_expression(left, value, time).time();\n                let right = self.exec_expression(right, value, time).interval();\n\n                let res: i64 = unsafe {\n                    pg_sys::DirectFunctionCall2Coll(\n                        Some($calc),\n                        pg_sys::InvalidOid,\n                        pg_sys::Datum::from(left),\n                        pg_sys::Datum::from(right),\n                    )\n                    .value() as _\n                };\n\n                Value::Time(res)\n            }};\n        }\n\n        match op {\n            // arithmetic operators\n            Plus => match ty {\n                Double => float_op!((left, right) left + right),\n                Time => time_op!((left, right) timestamptz_pl_interval),\n                Interval => interval_op!((left, right) interval_pl),\n                _ => unreachable!(),\n            },\n\n            Minus => match ty {\n                Double => float_op!((left, right) left - right),\n                Time => time_op!((left, right) timestamptz_mi_interval),\n                Interval => interval_op!((left, right) interval_mi),\n                _ => unreachable!(),\n            },\n\n            Mul => match ty {\n                Double => float_op!((left, right) left * right),\n                Interval => interval_float_op!((left, right) interval_mul),\n                _ => unreachable!(),\n            },\n\n            Div => match ty {\n                Double => float_op!((left, right) left / right),\n                Interval => interval_float_op!((left, right) interval_div),\n                _ => unreachable!(),\n            },\n\n            Pow => float_op!((left, right) left.powf(right)),\n\n            // comparison operators\n            Eq => {\n                let left = self.exec_expression(left, value, time);\n                let right = self.exec_expression(right, value, time);\n                (left == right).into()\n            }\n\n            Neq => {\n                let left = self.exec_expression(left, value, time);\n                let right = self.exec_expression(right, value, time);\n                (left != right).into()\n            }\n\n            Lt => {\n                let left = self.exec_expression(left, value, time);\n                let right = self.exec_expression(right, value, time);\n                (left < right).into()\n            }\n\n            Gt => {\n                let left = self.exec_expression(left, value, time);\n                let right = self.exec_expression(right, value, time);\n                (left > right).into()\n            }\n\n            Le => {\n                let left = self.exec_expression(left, value, time);\n                let right = self.exec_expression(right, value, time);\n                (left <= right).into()\n            }\n\n            Ge => {\n                let left = self.exec_expression(left, value, time);\n                let right = self.exec_expression(right, value, time);\n                (left >= right).into()\n            }\n\n            // boolean operators\n            And => {\n                let left = self.exec_expression(left, value, time).bool();\n                if !left {\n                    return false.into();\n                }\n                self.exec_expression(right, value, time)\n            }\n\n            Or => {\n                let left = self.exec_expression(left, value, time).bool();\n                if left {\n                    return true.into();\n                }\n                self.exec_expression(right, value, time)\n            }\n        }\n    }\n}\n\npub trait Tracer {\n    fn trace(&mut self, expr: &ExpressionSegment, result: &Value);\n}\n\nimpl Tracer for () {\n    fn trace(&mut self, _: &ExpressionSegment, _: &Value) {}\n}\n\nimpl<T> Tracer for T\nwhere\n    T: FnMut(&ExpressionSegment, &Value),\n{\n    fn trace(&mut self, expr: &ExpressionSegment, result: &Value) {\n        self(expr, result)\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/lambda/lambda_expr.pest",
    "content": "calculation = _{ SOI ~ let_expr ~ EOI }\nlet_expr = { (\"let\" ~ var ~ \"=\" ~ tuple ~ \";\")* ~ tuple }\ntuple = { binops ~ (\",\" ~ binops)* }\nbinops = { unary ~ (operation ~ unary)* }\nunary = _{ neg | not | term }\nneg = { \"-\" ~ unary }\nnot = { ^\"not\" ~ unary }\nterm = _{\n    val_var | time_var | var\n    | time | interval | num | function\n    | \"(\" ~ let_expr ~ \")\"\n}\nfunction = { function_name ~ \"(\" ~ (binops ~ (\",\" ~ binops)*  ~ \",\"?)? ~ \")\" }\n\noperation = _{\n    add | subtract | multiply | divide | power\n    | eq | neq | le | ge | lt | gt\n    | and | or\n}\n    add      = { \"+\" }\n    subtract = { \"-\" }\n    multiply = { \"*\" }\n    divide   = { \"/\" }\n    power    = { \"^\" }\n    eq       = { \"=\" }\n    neq      = { \"!=\" | \"<>\" }\n    lt       = { \"<\" }\n    le       = { \"<=\" }\n    gt       = { \">\" }\n    ge       = { \">=\" }\n    and      = { ^\"and\" }\n    or       = { ^\"or\" }\n\nnum = @{ int ~ (\".\" ~ ASCII_DIGIT*)? ~ (^\"e\" ~ int)? }\n    int = { (\"+\" | \"-\")? ~ ASCII_DIGIT+ }\n\ntime_var = @{ ^\"$time\" }\nval_var = @{ ^\"$value\" }\n\ntime = @{ string ~ \"t\" }\ninterval = @{ string ~ \"i\" }\nstring = _{ \"'\" ~ (!\"'\" ~ ANY)* ~ \"'\" }\n\nvar = @{ \"$\" ~ (ASCII_ALPHANUMERIC | \"_\")+ }\nfunction_name = @{ ASCII_ALPHA ~ ASCII_ALPHANUMERIC* }\n\nWHITESPACE = _{ \" \" | \"\\t\" | NEWLINE }\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/lambda/parser.rs",
    "content": "use std::{collections::HashMap, ffi::CString};\n\nuse pgrx::*;\n\nuse super::*;\n\nuse pest::{\n    iterators::{Pair, Pairs},\n    prec_climber::{Assoc, Operator, PrecClimber},\n    Parser,\n};\n\nuse ExpressionSegment::*;\nuse Rule::*;\nuse Type::*;\nuse UnaryOp::*;\n\n// Idealized expression grammar ignoring precedence\n// ```\n// Expression       :=  'let' Variable '=' Expression ';' Expression | BinaryExpression\n// BinaryExpression := PrefixExpression ({',', '+', '-', '*', ...}  BinaryExpression)\n// PrefixExpression := {'-', 'NOT'} ParenExpression\n// ParenExpression  := '(' Expression ')' | Variable | Literal\n// Variable         := $[a-bA-B_][a-bA-B0-9_]*\n// Literal          := <number> | '<string>'\n// ```\n// Josh - I believe this is unambiguous and LL(1), but we should check before\n//        stabilization\n// FIXME check the grammar\n\n#[derive(pest_derive::Parser)]\n#[grammar = \"time_vector/pipeline/lambda/lambda_expr.pest\"] // relative to src\npub struct ExpressionParser;\n\npub fn parse_expression(input: &str) -> Expression {\n    let parsed = ExpressionParser::parse(calculation, input).unwrap_or_else(|e| panic!(\"{}\", e));\n\n    let mut variables = Vec::new();\n    let expr = build_expression(parsed, &mut variables, &mut HashMap::new());\n    Expression { variables, expr }\n}\n\n// main parsing function.\nfn build_expression<'a>(\n    parsed: Pairs<'a, Rule>,\n    var_expressions: &mut Vec<ExpressionSegment>,\n    known_vars: &mut HashMap<&'a str, (Type, usize)>,\n) -> ExpressionSegment {\n    // Everything except binary operations are handled by `parse_primary()`\n    // when we encounter a sequence of binary operations eg `<> + <> * <>`\n    // the `(Expression, op, Expression)` triple is passed to `build_binary_op()`\n    // in descending precedence order.\n    PREC_CLIMBER.climb(\n        parsed,\n        |pair| parse_primary(pair, var_expressions, known_vars),\n        |left: ExpressionSegment, op: Pair<Rule>, right: ExpressionSegment| {\n            build_binary_op(op, left, right)\n        },\n    )\n}\n\n// handles everything except infix binary operators, which are handled by the\n// precedence climber and `build_binary_op()`\nfn parse_primary<'a>(\n    pair: Pair<'a, Rule>,\n    var_expressions: &mut Vec<ExpressionSegment>,\n    known_vars: &mut HashMap<&'a str, (Type, usize)>,\n) -> ExpressionSegment {\n    // HOW TO READ:\n    //   every rule (the left hand side of the `=` in the `.pest` file) has a\n    //   variant in the following `match` statement. When seeing a rule like\n    //   ```\n    //   foo = { bar ~ \"baz\" ~ qux }\n    //   ```\n    //   1. `pair.as_str()` will be the entire string that matched the rule.\n    //   2. `pair.into_iterator()` returns an iterator over the `Pair`s\n    //       representing the sub rules. In this case it'll return two, one for\n    //       `bar` and one for `qux`. These 'Pair's can be passed back to\n    //       `parse_primary()` to parse them into `Expression`s for further\n    //       handling.\n    match pair.as_rule() {\n        num => {\n            let val: f64 = pair.as_str().parse().unwrap();\n            DoubleConstant(val)\n        }\n\n        val_var => ValueVar,\n        time_var => TimeVar,\n\n        time => {\n            let s = pair.as_str();\n            let parsed_time = parse_timestamptz(&s[1..s.len() - 2]);\n            TimeConstant(parsed_time)\n        }\n\n        interval => {\n            let s = pair.as_str();\n            let parsed_interval = parse_interval(&s[1..s.len() - 2]);\n            IntervalConstant(parsed_interval)\n        }\n\n        var => {\n            let (ty, v) = known_vars\n                .get(pair.as_str())\n                .unwrap_or_else(|| panic!(\"unknown variable: {}\", pair.as_str()))\n                .clone();\n            UserVar(v, ty)\n        }\n\n        function => {\n            let mut pairs = pair.into_inner();\n            let func_name = pairs.next().unwrap();\n            let (num_args, func_id) = *BUILTIN_FUNCTION\n                .get(func_name.as_str())\n                .unwrap_or_else(|| panic!(\"unknown function: {}\", func_name.as_str()));\n\n            let args: Vec<_> = pairs\n                .map(|p| parse_primary(p, var_expressions, known_vars))\n                .collect();\n            if args.len() != num_args {\n                panic!(\n                    \"function `{}` expects {} arguments and received {}\",\n                    func_name.as_str(),\n                    num_args,\n                    args.len(),\n                )\n            }\n\n            FunctionCall(func_id, args)\n        }\n\n        neg => {\n            let value = pair.into_inner().next().unwrap();\n            let value = parse_primary(value, var_expressions, known_vars);\n            if value.ty() != &Double {\n                panic!(\"can only apply `-` to a DOUBLE PRECISION\")\n            }\n            Unary(Negative, value.into(), Double)\n        }\n\n        not => {\n            let value = pair.into_inner().next().unwrap();\n            let value = parse_primary(value, var_expressions, known_vars);\n            if value.ty() != &Bool {\n                panic!(\"can only apply NOT to a BOOLEAN\")\n            }\n            Unary(Not, value.into(), Bool)\n        }\n\n        // pass the sequence of binary operation to the precedence_climber to handle\n        binops => build_expression(pair.into_inner(), var_expressions, known_vars),\n\n        let_expr => {\n            let mut pairs = pair.into_inner();\n            loop {\n                // let_expr has two forms\n                // `let <variable> = <expression>; <expression>` and `<expression>`\n                // if we have more than one sub-pair in our pairs then we know we're\n                // in the first state, otherwise we must be in the second.\n                let var_name_or_expr = pairs.next().unwrap();\n                let var_value = match pairs.next() {\n                    None => return parse_primary(var_name_or_expr, var_expressions, known_vars),\n                    Some(val) => val,\n                };\n\n                let var_value = parse_primary(var_value, var_expressions, known_vars);\n\n                let var_name = var_name_or_expr.as_str();\n                known_vars\n                    .entry(var_name)\n                    .and_modify(|_| panic!(\"duplicate var {var_name}\"))\n                    .or_insert_with(|| (var_value.ty().clone(), var_expressions.len()));\n                var_expressions.push(var_value);\n            }\n        }\n\n        tuple => {\n            // the tuple rule effectively has two forms\n            // `<binops>` and `<binops> (, <binops>)+`\n            // it's only in the second case that we'll actually build something\n            // of a tuple type, in the former we'll just turn into the inner\n            // expression.\n            let mut pairs = pair.into_inner();\n            let first = pairs.next().unwrap();\n            let first_val = parse_primary(first, var_expressions, known_vars);\n            match pairs.next() {\n                None => first_val,\n                Some(pair) => {\n                    let mut vals = vec![first_val];\n                    let val = parse_primary(pair, var_expressions, known_vars);\n                    vals.push(val);\n                    for p in pairs {\n                        let val = parse_primary(p, var_expressions, known_vars);\n                        vals.push(val);\n                    }\n                    let ty = Tuple(vals.iter().map(|v| v.ty().clone()).collect());\n                    BuildTuple(vals, ty)\n                }\n            }\n        }\n\n        // operations marked with a `_` or that are below a `@` are never passed\n        // to us, so we can ignore them.\n        EOI | int | operation | string | unary | term | function_name | WHITESPACE\n        | calculation => unreachable!(\"{} should be transparent\", pair),\n\n        // infix operations should be passed to `build_binary_op()` by the\n        // precedence climber, so we should never see them here.\n        add | subtract | multiply | divide | power | eq | neq | lt | le | gt | ge | and | or => {\n            unreachable!(\"{} should be handled by precedence climbing\", pair)\n        }\n    }\n}\n\nfn build_binary_op(\n    op: Pair<Rule>,\n    left: ExpressionSegment,\n    right: ExpressionSegment,\n) -> ExpressionSegment {\n    use BinOp::*;\n    use Type::Interval;\n    macro_rules! return_ty {\n        ($op:literal $(($l: pat, $r:pat) => $ty:expr),+ $(,)?) => {\n            match (left.ty(), right.ty()) {\n                $(($l, $r) => $ty,)+\n                // TODO the error should report the location\n                (l, r) => panic!(\n                    concat!(\"no operator `{:?} {op} {:?}` only \",\n                        $(\"`\", stringify!($l), \" {op} \", stringify!($r), \"` \",)+\n                    ),\n                    l, r, op=$op),\n            }\n        };\n    }\n    match op.as_rule() {\n        add => {\n            let result_type = return_ty!(\"+\"\n                (Double, Double) => Double,\n                (Type::Time, Interval) => Type::Time,\n                (Interval, Interval) => Interval,\n            );\n            Binary(Plus, left.into(), right.into(), result_type)\n        }\n\n        subtract => {\n            let result_type = return_ty!(\"-\"\n                (Double, Double) => Double,\n                (Type::Time, Interval) => Type::Time,\n                (Interval, Interval) => Interval,\n            );\n            Binary(Minus, left.into(), right.into(), result_type)\n        }\n\n        multiply => match (left.ty(), right.ty()) {\n            (Double, Double) => Binary(Mul, left.into(), right.into(), Double),\n            (Interval, Double) => Binary(Mul, left.into(), right.into(), Interval),\n            // TODO right now BinOp(Mul, .., Interval) expects the interval on the left\n            //      and the double on the left. We could check in the executor which one\n            //      actually is, but it seems easier to just revers the value here if\n            //      they're in an unexpected order.\n            (Double, Interval) => Binary(Mul, right.into(), left.into(), Interval),\n            (l, r) => {\n                panic!(\"no operator `{l:?} * {r:?}` only `DOUBLE * DOUBLE` and `INTERVAL * FLOAT`\")\n            }\n        },\n\n        divide => {\n            let result_type = return_ty!(\"/\"\n                (Double, Double) => Double,\n                (Interval, Double) => Interval,\n            );\n            Binary(Div, left.into(), right.into(), result_type)\n        }\n\n        power => {\n            let result_type = return_ty!(\"^\"\n                (Double, Double) => Double,\n            );\n            Binary(Pow, left.into(), right.into(), result_type)\n        }\n\n        eq => {\n            if left.ty() != right.ty() {\n                panic!(\n                    \"mismatched types for `=`: {:?}, {:?}\",\n                    left.ty(),\n                    right.ty()\n                )\n            }\n            Binary(Eq, left.into(), right.into(), Bool)\n        }\n\n        neq => {\n            if left.ty() != right.ty() {\n                panic!(\n                    \"mismatched types for `!=`: {:?}, {:?}\",\n                    left.ty(),\n                    right.ty()\n                )\n            }\n            Binary(Neq, left.into(), right.into(), Bool)\n        }\n\n        lt => {\n            if left.ty() != right.ty() {\n                panic!(\n                    \"mismatched types for `<`: {:?}, {:?}\",\n                    left.ty(),\n                    right.ty()\n                )\n            }\n            Binary(Lt, left.into(), right.into(), Bool)\n        }\n\n        le => {\n            if left.ty() != right.ty() {\n                panic!(\n                    \"mismatched types for `<=`: {:?}, {:?}\",\n                    left.ty(),\n                    right.ty()\n                )\n            }\n            Binary(Le, left.into(), right.into(), Bool)\n        }\n\n        gt => {\n            if left.ty() != right.ty() {\n                panic!(\n                    \"mismatched types for `>`: {:?}, {:?}\",\n                    left.ty(),\n                    right.ty()\n                )\n            }\n            Binary(Gt, left.into(), right.into(), Bool)\n        }\n\n        ge => {\n            if left.ty() != right.ty() {\n                panic!(\n                    \"mismatched types for `>=`: {:?}, {:?}\",\n                    left.ty(),\n                    right.ty()\n                )\n            }\n            Binary(Ge, left.into(), right.into(), Bool)\n        }\n\n        and => {\n            let result_type = return_ty!(\"and\"\n                (Bool, Bool) => Bool,\n            );\n            Binary(And, left.into(), right.into(), result_type)\n        }\n\n        or => {\n            let result_type = return_ty!(\"or\"\n                (Bool, Bool) => Bool,\n            );\n            Binary(Or, left.into(), right.into(), result_type)\n        }\n\n        _ => unreachable!(),\n    }\n}\n\nfn parse_timestamptz(val: &str) -> i64 {\n    // FIXME pgrx wraps all functions in rust wrappers, which makes them\n    //       uncallable with DirectFunctionCall(). Is there a way to\n    //       export both?\n    unsafe extern \"C-unwind\" {\n        #[allow(improper_ctypes)]\n        fn timestamptz_in(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n    }\n\n    let cstr = CString::new(val).unwrap();\n    let parsed_time = unsafe {\n        pg_sys::DirectFunctionCall3Coll(\n            Some(timestamptz_in),\n            pg_sys::InvalidOid as _,\n            pg_sys::Datum::from(cstr.as_ptr()),\n            pg_sys::Datum::from(pg_sys::InvalidOid),\n            pg_sys::Datum::from(-1i32),\n        )\n    };\n    parsed_time.value() as _\n}\n\nfn parse_interval(val: &str) -> *mut pg_sys::Interval {\n    // FIXME pgrx wraps all functions in rust wrappers, which makes them\n    //       uncallable with DirectFunctionCall(). Is there a way to\n    //       export both?\n    unsafe extern \"C-unwind\" {\n        #[allow(improper_ctypes)]\n        fn interval_in(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n    }\n\n    let cstr = CString::new(val).unwrap();\n    let parsed_interval = unsafe {\n        pg_sys::DirectFunctionCall3Coll(\n            Some(interval_in),\n            pg_sys::InvalidOid as _,\n            pg_sys::Datum::from(cstr.as_ptr()),\n            pg_sys::Datum::from(pg_sys::InvalidOid),\n            pg_sys::Datum::from(-1i32),\n        )\n    };\n    parsed_interval.cast_mut_ptr()\n}\n\n// This static determines the precedence of infix operators\nstatic PREC_CLIMBER: once_cell::sync::Lazy<PrecClimber<Rule>> = once_cell::sync::Lazy::new(|| {\n    use Assoc::*;\n\n    // operators according to their precedence, ordered in a vector\n    // from lowest to highest. Multiple operators with the same precedence are\n    // joined with `|`\n    PrecClimber::new(vec![\n        Operator::new(or, Left),\n        Operator::new(and, Left),\n        Operator::new(eq, Left)\n            | Operator::new(neq, Left)\n            | Operator::new(lt, Left)\n            | Operator::new(le, Left)\n            | Operator::new(gt, Left)\n            | Operator::new(ge, Left),\n        Operator::new(add, Left) | Operator::new(subtract, Left),\n        Operator::new(multiply, Left) | Operator::new(divide, Left),\n        Operator::new(power, Right),\n    ])\n});\n\n// Table of builtin functions (all of them for now).\n// Maps function name to a tuple (num arguments, function identifier)\nstatic BUILTIN_FUNCTION: once_cell::sync::Lazy<HashMap<&str, (usize, Function)>> =\n    once_cell::sync::Lazy::new(|| {\n        use Function::*;\n        [\n            (\"abs\", (1, Abs)),\n            (\"cbrt\", (1, Cbrt)),\n            (\"ceil\", (1, Ceil)),\n            (\"floor\", (1, Floor)),\n            (\"ln\", (1, Ln)),\n            (\"log10\", (1, Log10)),\n            (\"log\", (2, Log)),\n            (\"pi\", (0, Pi)),\n            (\"round\", (1, Round)),\n            (\"sign\", (1, Sign)),\n            (\"sqrt\", (1, Sqrt)),\n            (\"trunc\", (1, Trunc)),\n            (\"acos\", (1, Acos)),\n            (\"asin\", (1, Asin)),\n            (\"atan\", (1, Atan)),\n            (\"atan2\", (2, Atan2)),\n            (\"cos\", (1, Cos)),\n            (\"sin\", (1, Sin)),\n            (\"tan\", (1, Tan)),\n            (\"sinh\", (1, Sinh)),\n            (\"cosh\", (1, Cosh)),\n            (\"tanh\", (1, Tanh)),\n            (\"asinh\", (1, Asinh)),\n            (\"acosh\", (1, Acosh)),\n            (\"atanh\", (1, Atanh)),\n        ]\n        .into_iter()\n        .collect()\n    });\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/lambda.rs",
    "content": "use std::borrow::Cow;\n\nuse pgrx::{\n    iter::{SetOfIterator, TableIterator},\n    *,\n};\n\nuse super::*;\n\npub use executor::ExpressionExecutor;\n\nmod executor;\nmod parser;\n\npub use self::toolkit_experimental::{Lambda, LambdaData};\n\n#[pg_schema]\npub mod toolkit_experimental {\n    pub(crate) use super::*;\n\n    //\n    // lambda type\n    //\n\n    pg_type! {\n        #[derive(Debug)]\n        struct Lambda<'input> {\n            len: u32,\n            string: [u8; self.len],\n        }\n    }\n}\n\nimpl<'input> InOutFuncs for Lambda<'input> {\n    fn output(&self, buffer: &mut StringInfo) {\n        use crate::serialization::{str_to_db_encoding, EncodedStr::*};\n\n        let stringified = std::str::from_utf8(self.string.as_slice()).unwrap();\n        match str_to_db_encoding(stringified) {\n            Utf8(s) => buffer.push_str(s),\n            Other(s) => buffer.push_bytes(s.to_bytes()),\n        }\n    }\n\n    fn input(input: &std::ffi::CStr) -> Self\n    where\n        Self: Sized,\n    {\n        use crate::serialization::str_from_db_encoding;\n\n        let s = str_from_db_encoding(input);\n        // validate the string\n        let _ = parser::parse_expression(s);\n        unsafe {\n            flatten! {\n                Lambda {\n                    len: s.len() as _,\n                    string: s.as_bytes().into(),\n                }\n            }\n        }\n    }\n}\n\nimpl<'a> LambdaData<'a> {\n    pub fn parse(&self) -> Expression {\n        parser::parse_expression(std::str::from_utf8(self.string.as_slice()).unwrap())\n    }\n}\n\n//\n// Direct lambda execution functions for testing\n//\n\n#[pg_extern(stable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn bool_lambda<'a>(\n    lambda: toolkit_experimental::Lambda<'a>,\n    time: crate::raw::TimestampTz,\n    value: f64,\n) -> bool {\n    let expression = lambda.parse();\n    if expression.expr.ty() != &Type::Bool {\n        panic!(\"invalid return type, must return a BOOLEAN for {expression:?}\")\n    }\n    let mut executor = ExpressionExecutor::new(&expression);\n    executor.exec(value, time.into()).bool()\n}\n\n#[pg_extern(stable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn f64_lambda<'a>(\n    lambda: toolkit_experimental::Lambda<'a>,\n    time: crate::raw::TimestampTz,\n    value: f64,\n) -> f64 {\n    let expression = lambda.parse();\n    if expression.expr.ty() != &Type::Double {\n        panic!(\"invalid return type, must return a DOUBLE PRECISION\")\n    }\n    let mut executor = ExpressionExecutor::new(&expression);\n    executor.exec(value, time.into()).float()\n}\n\n#[pg_extern(stable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn ttz_lambda<'a>(\n    lambda: toolkit_experimental::Lambda<'a>,\n    time: crate::raw::TimestampTz,\n    value: f64,\n) -> crate::raw::TimestampTz {\n    let expression = lambda.parse();\n    if expression.expr.ty() != &Type::Time {\n        panic!(\"invalid return type, must return a TimestampTZ\")\n    }\n    let mut executor = ExpressionExecutor::new(&expression);\n    executor.exec(value, time.into()).time().into()\n}\n\nuse crate::raw::Interval;\n#[pg_extern(stable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn interval_lambda<'a>(\n    lambda: toolkit_experimental::Lambda<'a>,\n    time: crate::raw::TimestampTz,\n    value: f64,\n) -> Interval {\n    let expression = lambda.parse();\n    if expression.expr.ty() != &Type::Interval {\n        panic!(\"invalid return type, must return a INTERVAL\")\n    }\n    let mut executor = ExpressionExecutor::new(&expression);\n    pg_sys::Datum::from(executor.exec(value, time.into()).interval()).into()\n}\n\n#[pg_extern(stable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn point_lambda<'a>(\n    lambda: toolkit_experimental::Lambda<'a>,\n    time: crate::raw::TimestampTz,\n    value: f64,\n) -> TableIterator<'static, (name!(time, crate::raw::TimestampTz), name!(value, f64))> {\n    let expression = lambda.parse();\n    if !expression.expr.ty_is_ts_point() {\n        panic!(\"invalid return type, must return a (TimestampTZ, DOUBLE PRECISION)\")\n    }\n\n    let mut executor = ExpressionExecutor::new(&expression);\n    let columns = match executor.exec(value, time.into()) {\n        Value::Tuple(columns) => columns,\n        _ => unreachable!(),\n    };\n    TableIterator::new(Some((columns[0].time().into(), columns[1].float())).into_iter())\n}\n\n#[pg_extern(stable, parallel_safe, schema = \"toolkit_experimental\")]\npub fn trace_lambda<'a>(\n    lambda: toolkit_experimental::Lambda<'a>,\n    time: crate::raw::TimestampTz,\n    value: f64,\n) -> SetOfIterator<'static, String> {\n    let expression = lambda.parse();\n\n    let mut trace: Vec<_> = vec![];\n    let mut executor = ExpressionExecutor::with_fn_tracer(&expression, |e, v| {\n        trace.push((e.name(), format!(\"{v:?}\")))\n    });\n\n    let _ = executor.exec(value, time.into());\n    let col1_size = trace.iter().map(|(e, _)| e.len()).max().unwrap_or(0);\n\n    SetOfIterator::new(\n        trace\n            .into_iter()\n            .map(move |(e, v)| format!(\"{e:>col1_size$}: {v:?}\")),\n    )\n}\n\n//\n// Common types across the parser and executor\n//\n\n// expressions\n#[derive(Debug)]\npub struct Expression {\n    variables: Vec<ExpressionSegment>,\n    expr: ExpressionSegment,\n}\n\n#[derive(Clone, Debug)]\npub enum ExpressionSegment {\n    ValueVar,\n    TimeVar,\n    DoubleConstant(f64),\n    TimeConstant(i64),\n    IntervalConstant(*mut pg_sys::Interval),\n    UserVar(usize, Type),\n    Unary(UnaryOp, Box<Self>, Type),\n    Binary(BinOp, Box<Self>, Box<Self>, Type),\n    FunctionCall(Function, Vec<Self>),\n    BuildTuple(Vec<Self>, Type),\n}\n\n#[derive(Clone, Copy, Debug)]\npub enum UnaryOp {\n    Not,\n    Negative,\n}\n\n#[derive(Clone, Copy, Debug)]\npub enum BinOp {\n    Plus,\n    Minus,\n    Mul,\n    Div,\n    Pow,\n    Eq,\n    Lt,\n    Le,\n    Gt,\n    Ge,\n    Neq,\n    And,\n    Or,\n}\n\n#[derive(Clone, Copy, Debug)]\npub enum Function {\n    Abs,\n    Cbrt,\n    Ceil,\n    Floor,\n    Ln,\n    Log10,\n    Log,\n    Pi,\n    Round,\n    Sign,\n    Sqrt,\n    Trunc,\n    Acos,\n    Asin,\n    Atan,\n    Atan2,\n    Cos,\n    Sin,\n    Tan,\n    Sinh,\n    Cosh,\n    Tanh,\n    Asinh,\n    Acosh,\n    Atanh,\n}\n\n// types\n#[derive(Clone, Debug, PartialEq, Eq)]\npub enum Type {\n    Time,\n    Double,\n    Bool,\n    Interval,\n    Tuple(Vec<Self>),\n}\n\n// values\n#[derive(Clone, Debug)]\npub enum Value {\n    Bool(bool),\n    Double(f64),\n    Time(i64),\n    Interval(*mut pg_sys::Interval),\n    Tuple(Vec<Self>),\n}\n\nimpl Expression {\n    pub fn ty(&self) -> &Type {\n        self.expr.ty()\n    }\n\n    pub fn ty_is_ts_point(&self) -> bool {\n        self.expr.ty_is_ts_point()\n    }\n}\n\nimpl ExpressionSegment {\n    pub fn ty(&self) -> &Type {\n        use ExpressionSegment::*;\n        use Type::*;\n        match self {\n            ValueVar => &Double,\n            TimeVar => &Time,\n            DoubleConstant(_) => &Double,\n            TimeConstant(_) => &Time,\n            IntervalConstant(_) => &Interval,\n            UserVar(_, ty) => ty,\n            FunctionCall(_, _) => &Double,\n            Unary(_, _, ty) => ty,\n            Binary(_, _, _, ty) => ty,\n            BuildTuple(_, ty) => ty,\n        }\n    }\n\n    pub fn ty_is_ts_point(&self) -> bool {\n        let columns = match self {\n            ExpressionSegment::BuildTuple(_, Type::Tuple(ty)) => ty,\n            _ => return false,\n        };\n\n        matches!(&**columns, [Type::Time, Type::Double])\n    }\n\n    pub fn name(&self) -> Cow<'static, str> {\n        use ExpressionSegment::*;\n        match self {\n            ValueVar => \"$value\".into(),\n            TimeVar => \"$time\".into(),\n            DoubleConstant(_) => \"f64 const\".into(),\n            TimeConstant(_) => \"time const\".into(),\n            IntervalConstant(_) => \"interval const\".into(),\n            UserVar(i, t) => format!(\"user var {i}: {t:?}\").into(),\n            Unary(op, _, t) => format!(\"uop {op:?} {t:?}\").into(),\n            Binary(op, _, _, t) => format!(\"binop {op:?} {t:?}\").into(),\n            FunctionCall(f, _) => format!(\"function {f:?}\").into(),\n            BuildTuple(_, t) => format!(\"tuple {t:?}\").into(),\n        }\n    }\n}\n\nimpl Value {\n    pub(crate) fn bool(&self) -> bool {\n        match self {\n            Value::Bool(b) => *b,\n            _ => unreachable!(),\n        }\n    }\n\n    pub(crate) fn float(&self) -> f64 {\n        match self {\n            Value::Double(f) => *f,\n            _ => unreachable!(),\n        }\n    }\n\n    pub(crate) fn time(&self) -> i64 {\n        match self {\n            Value::Time(t) => *t,\n            _ => unreachable!(),\n        }\n    }\n\n    pub(crate) fn interval(&self) -> *mut pg_sys::Interval {\n        match self {\n            Value::Interval(i) => *i,\n            _ => unreachable!(),\n        }\n    }\n}\n\nimpl PartialOrd for Value {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        use std::mem::discriminant;\n        use Value::*;\n\n        // XXX `NodeTag` somewhere inside `pg_sys::FunctionCallInfo` triggers\n        // `improper_ctypes` lint. The `pgrx` author explains the issue in\n        // details here:\n        //\n        // https://github.com/rust-lang/rust/issues/116831\n        //\n        // For now it seems OK to suppress these warnings here and below with\n        // #[allow(improper_ctypes)]\n        unsafe extern \"C-unwind\" {\n            #[allow(improper_ctypes)]\n            fn interval_cmp(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n        }\n\n        if discriminant(self) != discriminant(other) {\n            return None;\n        }\n        match (self, other) {\n            (Bool(l0), Bool(r0)) => l0.partial_cmp(r0),\n            (Double(l0), Double(r0)) => l0.partial_cmp(r0),\n            (Time(l0), Time(r0)) => l0.partial_cmp(r0),\n            (Tuple(l0), Tuple(r0)) => l0.partial_cmp(r0),\n            (Interval(l0), Interval(r0)) => unsafe {\n                let res = pg_sys::DirectFunctionCall2Coll(\n                    Some(interval_cmp),\n                    pg_sys::InvalidOid,\n                    pg_sys::Datum::from(*l0),\n                    pg_sys::Datum::from(*r0),\n                )\n                .value() as i32;\n                res.cmp(&0).into()\n            },\n            (_, _) => None,\n        }\n    }\n}\n\nimpl PartialEq for Value {\n    fn eq(&self, other: &Self) -> bool {\n        use std::mem::discriminant;\n        use Value::*;\n        unsafe extern \"C-unwind\" {\n            #[allow(improper_ctypes)]\n            fn interval_eq(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum;\n        }\n\n        if discriminant(self) != discriminant(other) {\n            return false;\n        }\n        match (self, other) {\n            (Bool(l0), Bool(r0)) => l0 == r0,\n            (Double(l0), Double(r0)) => l0 == r0,\n            (Time(l0), Time(r0)) => l0 == r0,\n            (Tuple(l0), Tuple(r0)) => l0 == r0,\n            (Interval(l0), Interval(r0)) => unsafe {\n                let res = pg_sys::DirectFunctionCall2Coll(\n                    Some(interval_eq),\n                    pg_sys::InvalidOid,\n                    pg_sys::Datum::from(*l0),\n                    pg_sys::Datum::from(*r0),\n                );\n                res.value() != 0\n            },\n            (_, _) => false,\n        }\n    }\n}\n\nimpl From<bool> for Value {\n    fn from(b: bool) -> Self {\n        Self::Bool(b)\n    }\n}\n\nimpl From<f64> for Value {\n    fn from(f: f64) -> Self {\n        Self::Double(f)\n    }\n}\n\nimpl<'a> Lambda<'a> {\n    pub fn into_data(self) -> LambdaData<'a> {\n        self.0\n    }\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    macro_rules! trace_lambda {\n        ($client: expr, $expr:literal) => {\n            $client\n                .update(\n                    concat!(\"SELECT trace_lambda($$ \", $expr, \" $$, '2021-01-01', 2.0)\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .map(|r| r.get::<String>(1).unwrap().unwrap())\n                .collect()\n        };\n    }\n\n    macro_rules! point_lambda {\n        ($client: expr, $expr:literal) => {\n            $client\n                .update(\n                    concat!(\n                        \"SELECT point_lambda($$ \",\n                        $expr,\n                        \" $$, '2021-01-01', 2.0)::text\"\n                    ),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    macro_rules! interval_lambda {\n        ($client: expr, $expr:literal) => {\n            $client\n                .update(\n                    concat!(\n                        \"SELECT interval_lambda($$ \",\n                        $expr,\n                        \" $$, now(), 2.0)::text\"\n                    ),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    macro_rules! f64_lambda {\n        ($client: expr, $expr:literal) => {\n            $client\n                .update(\n                    concat!(\"SELECT f64_lambda($$ \", $expr, \" $$, now(), 2.0)\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    macro_rules! bool_lambda {\n        ($client: expr, $expr:literal) => {\n            $client\n                .update(\n                    concat!(\"SELECT bool_lambda($$ \", $expr, \" $$, now(), 2.0)::text\"),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n\n    macro_rules! point_lambda_eq {\n        ($client: expr, $expr:literal, $expects:literal) => {\n            assert_eq!(point_lambda!($client, $expr), $expects,)\n        };\n    }\n\n    macro_rules! interval_lambda_eq {\n        ($client: expr, $expr:literal, $expects:literal) => {\n            assert_eq!(interval_lambda!($client, $expr), $expects,)\n        };\n    }\n\n    macro_rules! f64_lambda_eq {\n        ($client: expr, $expr:literal, $expects:expr) => {\n            assert!((f64_lambda!($client, $expr) - ($expects)).abs() < f64::EPSILON,)\n        };\n    }\n\n    macro_rules! bool_lambda_eq {\n        ($client: expr, $expr:literal, $expects:literal) => {\n            assert_eq!(bool_lambda!($client, $expr), $expects,)\n        };\n    }\n\n    #[pg_test]\n    fn test_lambda_general() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n            client\n                .update(\n                    \"SELECT $$ let $1 = 1.0; 2.0, $1 $$::toolkit_experimental.lambda\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            // client.update(\"SELECT $$ '1 day'i $$::toolkit_experimental.lambda\", None, &[]).unwrap();\n            // client.update(\"SELECT $$ '2020-01-01't $$::toolkit_experimental.lambda\", None, &[]).unwrap();\n\n            let res = client\n                .update(\"SELECT f64_lambda($$ 1.0 $$, now(), 0.0)::text\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"1\");\n\n            let res = client\n                .update(\n                    \"SELECT f64_lambda($$ 1.0 + 1.0 $$, now(), 0.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"2\");\n\n            let res = client\n                .update(\n                    \"SELECT f64_lambda($$ 1.0 - 1.0 $$, now(), 0.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"0\");\n\n            let res = client\n                .update(\n                    \"SELECT f64_lambda($$ 2.0 * 3.0 $$, now(), 0.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"6\");\n\n            let res = client\n                .update(\n                    \"SELECT f64_lambda($$ $value + 3.0 $$, now(), 2.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"5\");\n\n            let res = client\n                .update(\n                    \"SELECT f64_lambda($$ 3.0 - 1.0 * 3.0 $$, now(), 2.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"0\");\n\n            bool_lambda_eq!(client, \"3.0 = 3.0\", \"true\");\n            bool_lambda_eq!(client, \"3.0 != 3.0\", \"false\");\n            bool_lambda_eq!(client, \"2.0 != 3.0\", \"true\");\n            bool_lambda_eq!(client, \"2.0 != 3.0 and 1 = 1\", \"true\");\n            bool_lambda_eq!(client, \"2.0 != 3.0 and (1 = 1)\", \"true\");\n\n            let res = client\n                .update(\n                    \"SELECT ttz_lambda($$ '2020-11-22 13:00:01't $$, now(), 2.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"2020-11-22 13:00:01+00\");\n\n            let res = client\n                .update(\n                    \"SELECT ttz_lambda($$ $time $$, '1930-01-12 14:20:21', 2.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"1930-01-12 14:20:21+00\");\n\n            let res = client\n                .update(\n                    \"SELECT ttz_lambda($$ '2020-11-22 13:00:01't - '1 day'i $$, now(), 2.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"2020-11-21 13:00:01+00\");\n\n            let res = client\n                .update(\n                    \"SELECT ttz_lambda($$ '2020-11-22 13:00:01't + '1 day'i $$, now(), 2.0)::text\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(&*res.unwrap(), \"2020-11-23 13:00:01+00\");\n\n            point_lambda_eq!(\n                client,\n                \"'2020-11-22 13:00:01't + '1 day'i, 2.0 * 3.0\",\n                r#\"(\"2020-11-23 13:00:01+00\",6)\"#\n            );\n\n            point_lambda_eq!(\n                client,\n                \"($time, $value^2 + $value * 2.3 + 43.2)\",\n                r#\"(\"2021-01-01 00:00:00+00\",51.800000000000004)\"#\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_lambda_comparison() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            bool_lambda_eq!(client, \"2.0 <  3.0\", \"true\");\n            bool_lambda_eq!(client, \"2.0 <= 3.0\", \"true\");\n            bool_lambda_eq!(client, \"2.0 >  3.0\", \"false\");\n            bool_lambda_eq!(client, \"2.0 >= 3.0\", \"false\");\n            bool_lambda_eq!(client, \"4.0 >  3.0\", \"true\");\n            bool_lambda_eq!(client, \"4.0 >= 3.0\", \"true\");\n            bool_lambda_eq!(client, \"4.0 >  4.0\", \"false\");\n            bool_lambda_eq!(client, \"4.0 >= 4.0\", \"true\");\n\n            bool_lambda_eq!(client, \"'2020-01-01't <  '2021-01-01't\", \"true\");\n            bool_lambda_eq!(client, \"'2020-01-01't <= '2021-01-01't\", \"true\");\n            bool_lambda_eq!(client, \"'2020-01-01't >  '2021-01-01't\", \"false\");\n            bool_lambda_eq!(client, \"'2020-01-01't >= '2021-01-01't\", \"false\");\n            bool_lambda_eq!(client, \"'2022-01-01't <  '2021-01-01't\", \"false\");\n            bool_lambda_eq!(client, \"'2022-01-01't <= '2021-01-01't\", \"false\");\n            bool_lambda_eq!(client, \"'2022-01-01't >  '2021-01-01't\", \"true\");\n            bool_lambda_eq!(client, \"'2022-01-01't >= '2021-01-01't\", \"true\");\n            bool_lambda_eq!(client, \"'2022-01-01't >  '2021-01-01't\", \"true\");\n            bool_lambda_eq!(client, \"'2022-01-01't >= '2021-01-01't\", \"true\");\n\n            bool_lambda_eq!(client, \"'1 day'i  <  '1 week'i\", \"true\");\n            bool_lambda_eq!(client, \"'1 day'i  <= '1 week'i\", \"true\");\n            bool_lambda_eq!(client, \"'1 day'i  >  '1 week'i\", \"false\");\n            bool_lambda_eq!(client, \"'1 day'i  >= '1 week'i \", \"false\");\n            bool_lambda_eq!(client, \"'1 year'i >  '1 week'i\", \"true\");\n            bool_lambda_eq!(client, \"'1 year'i >= '1 week'i\", \"true\");\n            bool_lambda_eq!(client, \"'1 year'i >  '1 year'i\", \"false\");\n            bool_lambda_eq!(client, \"'1 year'i >= '1 year'i\", \"true\");\n        });\n    }\n\n    #[pg_test]\n    fn test_lambda_function() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            f64_lambda_eq!(client, \"pi()\", std::f64::consts::PI);\n\n            f64_lambda_eq!(client, \"abs(-2.0)\", (-2.0f64).abs());\n            f64_lambda_eq!(client, \"cbrt(-2.0)\", (-2.0f64).cbrt());\n            f64_lambda_eq!(client, \"ceil(-2.1)\", (-2.1f64).ceil());\n            f64_lambda_eq!(client, \"floor(-2.1)\", (-2.1f64).floor());\n            f64_lambda_eq!(client, \"ln(2.0)\", (2.0f64).ln());\n            f64_lambda_eq!(client, \"log10(2.0)\", (2.0f64).log10());\n            f64_lambda_eq!(client, \"round(-2.1)\", (-2.1f64).round());\n            f64_lambda_eq!(client, \"sign(-2.0)\", (-2.0f64).signum());\n            f64_lambda_eq!(client, \"sqrt(2.0)\", (2.0f64).sqrt());\n            f64_lambda_eq!(client, \"trunc(-2.0)\", (-2.0f64).trunc());\n            f64_lambda_eq!(client, \"acos(0.2)\", (0.2f64).acos());\n            f64_lambda_eq!(client, \"asin(0.2)\", (0.2f64).asin());\n            f64_lambda_eq!(client, \"atan(0.2)\", (0.2f64).atan());\n            f64_lambda_eq!(client, \"cos(2.0)\", (2.0f64).cos());\n            f64_lambda_eq!(client, \"sin(2.0)\", (2.0f64).sin());\n            f64_lambda_eq!(client, \"tan(2.0)\", (2.0f64).tan());\n            f64_lambda_eq!(client, \"sinh(2.0)\", (2.0f64).sinh());\n            f64_lambda_eq!(client, \"cosh(2.0)\", (2.0f64).cosh());\n            f64_lambda_eq!(client, \"tanh(2.0)\", (2.0f64).tanh());\n            f64_lambda_eq!(client, \"asinh(1.0)\", (1.0f64).asinh());\n            f64_lambda_eq!(client, \"acosh(1.0)\", (1.0f64).acosh());\n            f64_lambda_eq!(client, \"atanh(0.9)\", (0.9f64).atanh());\n\n            f64_lambda_eq!(client, \"log(2.0, 10)\", 2.0f64.log(10.0));\n            f64_lambda_eq!(client, \"atan2(2.0, 10)\", 2.0f64.atan2(10.0));\n        });\n    }\n\n    #[pg_test]\n    fn test_lambda_unary() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            f64_lambda_eq!(client, \"-(2.0)\", -2.0f64);\n            f64_lambda_eq!(client, \"-(-2.0)\", 2.0f64);\n\n            bool_lambda_eq!(client, \"not (1 = 1)\", \"false\");\n            bool_lambda_eq!(client, \"not (1 = 2)\", \"true\");\n            bool_lambda_eq!(client, \"not not (1 = 1)\", \"true\");\n            bool_lambda_eq!(client, \"not not (1 = 2)\", \"false\");\n            bool_lambda_eq!(client, \"not (1 <> 1)\", \"true\");\n            bool_lambda_eq!(client, \"not (1 <> 2)\", \"false\");\n        });\n    }\n\n    #[pg_test]\n    fn test_lambda_interval_ops() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            interval_lambda_eq!(client, \"'1 day'i + '1 day'i\", \"2 days\");\n            interval_lambda_eq!(client, \"'1 day'i + '1 week'i\", \"8 days\");\n            interval_lambda_eq!(client, \"'1 week'i - '1 day'i\", \"6 days\");\n\n            interval_lambda_eq!(client, \"'1 day'i * 3\", \"3 days\");\n            interval_lambda_eq!(client, \"4 * '1 day'i\", \"4 days\");\n            interval_lambda_eq!(client, \"'4 day'i / 4\", \"1 day\");\n        });\n    }\n\n    #[pg_test]\n    fn test_lambda_variable() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            f64_lambda_eq!(client, \"let $foo = 2.0; $foo\", 2.0);\n            f64_lambda_eq!(client, \"let $foo = -2.0; $foo\", -2.0);\n            f64_lambda_eq!(client, \"let $foo = abs(-2.0); $foo\", 2.0);\n            f64_lambda_eq!(client, \"let $foo = abs(-2.0); $foo * $foo\", 4.0);\n\n            bool_lambda_eq!(client, \"let $foo = 1 = 1; $foo\", \"true\");\n            bool_lambda_eq!(client, \"let $foo = 1 = 1; $foo and $foo\", \"true\");\n            bool_lambda_eq!(client, \"let $foo = 1 = 1; $foo or $foo\", \"true\");\n\n            // verify that variables are only expanded once\n            let rows: Vec<_> = trace_lambda!(client, \"let $bar = 1 + 1; $bar + $bar + $bar\");\n            assert_eq!(\n                &*rows,\n                [\n                    r#\"         f64 const: \"Double(1.0)\"\"#,\n                    r#\"         f64 const: \"Double(1.0)\"\"#,\n                    r#\" binop Plus Double: \"Double(2.0)\"\"#,\n                    r#\"user var 0: Double: \"Double(2.0)\"\"#,\n                    r#\"user var 0: Double: \"Double(2.0)\"\"#,\n                    r#\" binop Plus Double: \"Double(4.0)\"\"#,\n                    r#\"user var 0: Double: \"Double(2.0)\"\"#,\n                    r#\" binop Plus Double: \"Double(6.0)\"\"#,\n                ],\n            );\n\n            let rows: Vec<_> = trace_lambda!(\n                client,\n                \"let $foo = -2;\\nlet $bar = $foo * $foo;\\n $bar * $bar\"\n            );\n            assert_eq!(\n                &*rows,\n                [\n                    // TODO try and fix parsing so than `-2` parses as a constant `-2`\n                    r#\"          f64 const: \"Double(2.0)\"\"#,\n                    r#\"uop Negative Double: \"Double(-2.0)\"\"#,\n                    r#\" user var 0: Double: \"Double(-2.0)\"\"#,\n                    r#\" user var 0: Double: \"Double(-2.0)\"\"#,\n                    r#\"   binop Mul Double: \"Double(4.0)\"\"#,\n                    r#\" user var 1: Double: \"Double(4.0)\"\"#,\n                    r#\" user var 1: Double: \"Double(4.0)\"\"#,\n                    r#\"   binop Mul Double: \"Double(16.0)\"\"#,\n                ],\n            );\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/map.rs",
    "content": "use std::{\n    mem::{self, ManuallyDrop, MaybeUninit},\n    ptr,\n};\n\nuse pgrx::*;\n\nuse super::*;\n\nuse crate::serialization::PgProcId;\n\n// TODO is (stable, parallel_safe) correct?\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"map\",\n    schema = \"toolkit_experimental\"\n)]\npub fn map_lambda_pipeline_element<'l>(\n    lambda: toolkit_experimental::Lambda<'l>,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    let expression = lambda.parse();\n    if expression.ty() != &lambda::Type::Double && !expression.ty_is_ts_point() {\n        panic!(\"invalid lambda type: the lambda must return a DOUBLE PRECISION or (TimestampTZ, DOUBLE PRECISION)\")\n    }\n\n    Element::MapLambda {\n        lambda: lambda.into_data(),\n    }\n    .flatten()\n}\n\npub fn apply_lambda_to<'a>(\n    mut series: Timevector_TSTZ_F64<'a>,\n    lambda: &lambda::LambdaData<'_>,\n) -> Timevector_TSTZ_F64<'a> {\n    let expression = lambda.parse();\n    let only_val = expression.ty() == &lambda::Type::Double;\n    if !only_val && !expression.ty_is_ts_point() {\n        panic!(\"invalid lambda type: the lambda must return a DOUBLE PRECISION or (TimestampTZ, DOUBLE PRECISION)\")\n    }\n\n    let mut executor = lambda::ExpressionExecutor::new(&expression);\n\n    let invoke = |time: i64, value: f64| {\n        use lambda::Value::*;\n        executor.reset();\n        let result = executor.exec(value, time);\n        match result {\n            Double(f) => (None, Some(f)),\n            Time(t) => (Some(t), None),\n            Tuple(cols) => match &*cols {\n                [Time(t), Double(f)] => (Some(*t), Some(*f)),\n                _ => unreachable!(),\n            },\n\n            _ => unreachable!(),\n        }\n    };\n\n    map_lambda_over_series(&mut series, only_val, invoke);\n    series\n}\n\npub fn map_lambda_over_series(\n    series: &mut Timevector_TSTZ_F64<'_>,\n    only_val: bool,\n    mut func: impl FnMut(i64, f64) -> (Option<i64>, Option<f64>),\n) {\n    for point in series.points.as_owned() {\n        let (new_time, new_val) = func(point.ts, point.val);\n        *point = TSPoint {\n            ts: if only_val {\n                point.ts\n            } else {\n                new_time.unwrap_or(point.ts)\n            },\n            val: new_val.unwrap_or(point.val),\n        }\n    }\n}\n\n#[pg_extern(\n    stable,\n    parallel_safe,\n    name = \"map_series\",\n    schema = \"toolkit_experimental\"\n)]\npub fn map_series_pipeline_element(\n    function: crate::raw::regproc,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    map_series_element(crate::raw::regproc::from(function.0)).flatten()\n}\n\npub fn map_series_element<'a>(function: crate::raw::regproc) -> Element<'a> {\n    let function: pg_sys::regproc = pg_sys::Oid::from(function.0.value() as u32)\n        .try_into()\n        .unwrap();\n    check_user_function_type(function);\n    Element::MapSeries {\n        function: PgProcId(function),\n    }\n}\n\npub fn check_user_function_type(function: pg_sys::regproc) {\n    let mut argtypes: *mut pg_sys::Oid = ptr::null_mut();\n    let mut nargs: ::std::os::raw::c_int = 0;\n    let rettype = unsafe { pg_sys::get_func_signature(function, &mut argtypes, &mut nargs) };\n\n    if nargs != 1 {\n        error!(\"invalid number of mapping function arguments, expected fn(timevector) RETURNS timevector\")\n    }\n\n    assert!(!argtypes.is_null());\n    if unsafe { *argtypes } != *crate::time_vector::TIMEVECTOR_OID {\n        error!(\"invalid argument type, expected fn(timevector) RETURNS timevector\")\n    }\n\n    if rettype != *crate::time_vector::TIMEVECTOR_OID {\n        error!(\"invalid return type, expected fn(timevector) RETURNS timevector\")\n    }\n}\n\npub fn apply_to_series(\n    mut series: Timevector_TSTZ_F64<'_>,\n    func: pg_sys::RegProcedure,\n) -> Timevector_TSTZ_F64<'_> {\n    let mut flinfo: pg_sys::FmgrInfo = unsafe { MaybeUninit::zeroed().assume_init() };\n    unsafe {\n        pg_sys::fmgr_info(func, &mut flinfo);\n    };\n\n    unsafe {\n        // use pg_sys::FunctionCall1Coll to get the pg_guard\n        let res = pg_sys::FunctionCall1Coll(\n            &mut flinfo,\n            pg_sys::InvalidOid,\n            // SAFETY the input memory context will not end in the sub-function\n            //        and the sub-function will allocate the returned timevector\n            series.cached_datum_or_flatten(),\n        );\n        Timevector_TSTZ_F64::from_polymorphic_datum(res, false, pg_sys::InvalidOid)\n            .expect(\"unexpected NULL in timevector mapping function\")\n    }\n}\n\n// TODO is (stable, parallel_safe) correct?\n#[pg_extern(\n    stable,\n    parallel_safe,\n    name = \"map_data\",\n    schema = \"toolkit_experimental\"\n)]\npub fn map_data_pipeline_element(\n    function: crate::raw::regproc,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    let mut argtypes: *mut pg_sys::Oid = ptr::null_mut();\n    let mut nargs: ::std::os::raw::c_int = 0;\n    let rettype = unsafe {\n        pg_sys::get_func_signature(\n            pg_sys::Oid::from(function.0.value() as u32),\n            &mut argtypes,\n            &mut nargs,\n        )\n    };\n\n    if nargs != 1 {\n        error!(\"invalid number of mapping function arguments, expected fn(double precision) RETURNS double precision\")\n    }\n\n    if unsafe { *argtypes } != pgrx::PgBuiltInOids::FLOAT8OID.value() {\n        error!(\"invalid argument type, expected fn(double precision) RETURNS double precision\")\n    }\n\n    if rettype != pgrx::PgBuiltInOids::FLOAT8OID.value() {\n        error!(\"invalid return type, expected fn(double precision) RETURNS double precision\")\n    }\n\n    Element::MapData {\n        function: PgProcId(pg_sys::Oid::from(function.0.value() as u32)),\n    }\n    .flatten()\n}\n\npub fn apply_to(\n    mut series: Timevector_TSTZ_F64<'_>,\n    func: pg_sys::RegProcedure,\n) -> Timevector_TSTZ_F64<'_> {\n    let mut flinfo: pg_sys::FmgrInfo = unsafe { MaybeUninit::zeroed().assume_init() };\n\n    let fn_addr: unsafe extern \"C-unwind\" fn(\n        *mut pg_sys::FunctionCallInfoBaseData,\n    ) -> pg_sys::Datum;\n    let mut fc_info = unsafe {\n        pg_sys::fmgr_info(func, &mut flinfo);\n        fn_addr = flinfo.fn_addr.expect(\"null function in timevector map\");\n        union FcInfo1 {\n            data: ManuallyDrop<pg_sys::FunctionCallInfoBaseData>,\n            #[allow(dead_code)]\n            bytes: [u8; mem::size_of::<pg_sys::FunctionCallInfoBaseData>()\n                + mem::size_of::<pg_sys::NullableDatum>()],\n        }\n        FcInfo1 {\n            data: ManuallyDrop::new(pg_sys::FunctionCallInfoBaseData {\n                flinfo: &mut flinfo,\n                context: std::ptr::null_mut(),\n                resultinfo: std::ptr::null_mut(),\n                fncollation: pg_sys::InvalidOid,\n                isnull: false,\n                nargs: 1,\n                args: pg_sys::__IncompleteArrayField::new(),\n            }),\n        }\n    };\n\n    let invoke = |val: f64| unsafe {\n        let fc_info = &mut *fc_info.data;\n        let args = fc_info.args.as_mut_slice(1);\n        args[0].value = val.into_datum().unwrap();\n        args[0].isnull = false;\n        let res = fn_addr(fc_info);\n        f64::from_polymorphic_datum(res, fc_info.isnull, pg_sys::InvalidOid)\n            .expect(\"unexpected NULL in timevector mapping function\")\n    };\n\n    map_series(&mut series, invoke);\n    series\n}\n\npub fn map_series(series: &mut Timevector_TSTZ_F64<'_>, mut func: impl FnMut(f64) -> f64) {\n    use std::panic::AssertUnwindSafe;\n\n    let points = series.points.as_owned().iter_mut();\n    // setjump guard around the loop to reduce the amount we have to\n    // call it\n    // NOTE need to be careful that there's not allocation within the\n    //      loop body so it cannot leak\n    pg_sys::PgTryBuilder::new(AssertUnwindSafe(|| {\n        for point in points {\n            *point = TSPoint {\n                ts: point.ts,\n                val: func(point.val),\n            }\n        }\n    }))\n    .execute()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_pipeline_map_lambda() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client.update(\n                \"SELECT (timevector(time, value) -> map($$ ($time + '1 day'i, $value * 2) $$))::TEXT FROM series\",\n                None,\n                &[]\n            )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:50),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-06 00:00:00+00\\\",val:60)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_pipeline_map_lambda2() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            let expected = \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:725.7),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:166.2),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:489.2),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:302.7),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:1012.2)\\\n            ],null_val:[0])\";\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value) \\\n                    -> map($$ ($time, $value^2 + $value * 2.3 + 43.2) $$))::TEXT \\\n                    FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), expected);\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value) \\\n                    -> map($$ ($value^2 + $value * 2.3 + 43.2) $$))::TEXT \\\n                    FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(val.unwrap(), expected);\n        });\n    }\n\n    #[pg_test]\n    fn test_pipeline_map_data() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            client.update(\n                \"CREATE FUNCTION x2(double precision) RETURNS DOUBLE PRECISION AS 'SELECT $1 * 2;' LANGUAGE SQL\",\n                None,\n                &[]\n            ).unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value) -> map_data('x2'))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:50),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:40),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:60)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_pipeline_map_series() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            client.update(\n                \"CREATE FUNCTION jan_3_x3(timevector_tstz_f64) RETURNS timevector_tstz_f64 AS $$\\\n                    SELECT timevector(time, value * 3) \\\n                    FROM (SELECT (unnest($1)).*) a \\\n                    WHERE time='2020-01-03 00:00:00+00';\\\n                $$ LANGUAGE SQL\",\n                None,\n                &[],\n            ).unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value) -> map_series('jan_3_x3'))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:1,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:60)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n\n    #[pg_test]\n    #[should_panic = \"division by zero\"]\n    fn test_pipeline_map_series_failure() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\n                \"CREATE FUNCTION always_fail(timevector_tstz_f64) RETURNS timevector_tstz_f64 AS\n                $$\n                    SELECT 0/0;\n                    SELECT $1;\n                $$ LANGUAGE SQL\",\n                None,\n                &[],\n            ).unwrap();\n\n            client\n                .update(\n                    \"SELECT (timevector(time, value) -> map_series('always_fail'))::TEXT FROM series\",\n                    None,\n                    &[]\n                )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n        });\n    }\n\n    #[pg_test]\n    #[should_panic = \" returned NULL\"]\n    fn test_pipeline_map_series_null() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\n                \"CREATE FUNCTION always_null(timevector_tstz_f64) RETURNS timevector_tstz_f64 AS\n                $$\n                    SELECT NULL::timevector_tstz_f64;\n                $$ LANGUAGE SQL\",\n                None,\n                &[],\n            ).unwrap();\n\n            client\n                .update(\n                    \"SELECT (timevector(time, value) -> map_series('always_null'))::TEXT FROM series\",\n                    None,\n                    &[]\n                )\n                .unwrap().first()\n                .get_one::<String>().unwrap();\n        });\n    }\n\n    #[pg_test]\n    fn test_map_io() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25.0), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10.0), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20.0), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15.0), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30.0)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:5,flags:0,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[0])\"\n            );\n\n            client\n                .update(\n                    \"CREATE FUNCTION serier(timevector_tstz_f64) RETURNS timevector_tstz_f64 AS $$\\\n                    SELECT $1;\\\n                $$ LANGUAGE SQL\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE FUNCTION dater(double precision) RETURNS double precision AS $$\\\n                    SELECT $1 * 3;\\\n                $$ LANGUAGE SQL\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (a, b) = client\n                .update(\n                    \"SELECT map_series('serier')::TEXT, map_data('dater')::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<String, String>()\n                .unwrap();\n            let one = \"\\\n            (\\\n                version:1,\\\n                num_elements:1,\\\n                elements:[\\\n                    MapSeries(\\\n                        function:\\\"public.serier(public.timevector_tstz_f64)\\\"\\\n                    )\\\n                ]\\\n            )\";\n            let two = \"\\\n            (\\\n                version:1,\\\n                num_elements:1,\\\n                elements:[\\\n                    MapData(\\\n                        function:\\\"public.dater(double precision)\\\"\\\n                    )\\\n                ]\\\n            )\";\n            assert_eq!((&*a.unwrap(), &*b.unwrap()), (one, two));\n\n            // FIXME this doesn't work yet\n            let (a, b) = client\n                .update(\n                    &format!(\n                        \"SELECT \\\n                    '{one}'::UnstableTimevectorPipeline::Text, \\\n                    '{two}'::UnstableTimevectorPipeline::Text\"\n                    ),\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<String, String>()\n                .unwrap();\n            assert_eq!((&*a.unwrap(), &*b.unwrap()), (one, two));\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline/sort.rs",
    "content": "use pgrx::*;\n\nuse super::*;\n\n// TODO is (immutable, parallel_safe) correct?\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"sort\",\n    schema = \"toolkit_experimental\"\n)]\npub fn sort_pipeline_element<'p>() -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Element::Sort {}.flatten()\n}\n\npub fn sort_timevector(mut series: Timevector_TSTZ_F64<'_>) -> Timevector_TSTZ_F64<'_> {\n    if series.is_sorted() {\n        return series;\n    }\n\n    let (points, null_val) = if !series.has_nulls() {\n        // easy case\n        let mut points = std::mem::take(series.points.as_owned());\n        points.sort_by(|a, b| a.ts.cmp(&b.ts));\n        let nulls_len = points.len().div_ceil(8);\n        (points, std::vec::from_elem(0_u8, nulls_len))\n    } else {\n        let mut points: Vec<(usize, TSPoint)> = std::mem::take(series.points.as_owned())\n            .into_iter()\n            .enumerate()\n            .collect();\n        points.sort_by(|(_, a), (_, b)| a.ts.cmp(&b.ts));\n        let mut null_val = std::vec::from_elem(0_u8, points.len().div_ceil(8));\n        let points = points\n            .into_iter()\n            .enumerate()\n            .map(|(new_idx, (old_idx, ts))| {\n                if series.is_null_val(old_idx) {\n                    null_val[new_idx / 8] |= 1 << (new_idx % 8);\n                }\n                ts\n            })\n            .collect();\n        (points, null_val)\n    };\n\n    Timevector_TSTZ_F64Data {\n        header: 0,\n        version: 1,\n        padding: [0; 3],\n        num_points: points.len() as u32,\n        flags: series.flags | FLAG_IS_SORTED,\n        internal_padding: [0; 3],\n        points: points.into(),\n        null_val: null_val.into(),\n    }\n    .into()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_pipeline_sort() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE series(time timestamptz, value double precision)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO series \\\n                    VALUES \\\n                    ('2020-01-04 UTC'::TIMESTAMPTZ, 25), \\\n                    ('2020-01-01 UTC'::TIMESTAMPTZ, 10), \\\n                    ('2020-01-03 UTC'::TIMESTAMPTZ, 20), \\\n                    ('2020-01-02 UTC'::TIMESTAMPTZ, 15), \\\n                    ('2020-01-05 UTC'::TIMESTAMPTZ, 30), \\\n                    ('2020-01-02 12:00:00 UTC'::TIMESTAMPTZ, NULL)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value))::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:6,flags:2,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30),\\\n                (ts:\\\"2020-01-02 12:00:00+00\\\",val:NaN)\\\n            ],null_val:[32])\"\n            );\n\n            let val = client\n                .update(\n                    \"SELECT (timevector(time, value) -> sort())::TEXT FROM series\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:6,flags:3,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                (ts:\\\"2020-01-02 00:00:00+00\\\",val:15),\\\n                (ts:\\\"2020-01-02 12:00:00+00\\\",val:NaN),\\\n                (ts:\\\"2020-01-03 00:00:00+00\\\",val:20),\\\n                (ts:\\\"2020-01-04 00:00:00+00\\\",val:25),\\\n                (ts:\\\"2020-01-05 00:00:00+00\\\",val:30)\\\n            ],null_val:[4])\"\n            );\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector/pipeline.rs",
    "content": "mod aggregation;\nmod arithmetic;\nmod delta;\nmod expansion;\nmod fill_to;\nmod filter;\nmod lambda;\nmod map;\nmod sort;\n\nuse std::convert::TryInto;\n\nuse pgrx::*;\n\nuse super::*;\n\nuse crate::{flatten, pg_type, ron_inout_funcs};\n\nuse fill_to::{fill_to, FillToMethod};\n\nuse delta::timevector_delta;\nuse sort::sort_timevector;\n\npub use self::toolkit_experimental::*;\nuse crate::serialization::PgProcId;\n\n#[pg_schema]\npub mod toolkit_experimental {\n    use super::*;\n    pub use crate::time_vector::Timevector_TSTZ_F64;\n    pub(crate) use lambda::toolkit_experimental::{Lambda, LambdaData};\n    // TODO once we start stabilizing elements, create a type TimevectorPipeline\n    //      stable elements will create a stable pipeline, but adding an unstable\n    //      element to a stable pipeline will create an unstable pipeline\n    pg_type! {\n        #[derive(Debug)]\n        struct UnstableTimevectorPipeline<'input> {\n            num_elements: u64,\n            elements: [Element<'input>; self.num_elements],\n        }\n    }\n\n    flat_serialize_macro::flat_serialize! {\n        #[derive(Debug)]\n        #[derive(serde::Serialize, serde::Deserialize)]\n        enum Element<'input> {\n            kind: u64,\n            LTTB: 1 {\n                resolution: u64,\n            },\n            // 2 was for resample_to_rate\n            // 3 was for fill_holes\n            Sort: 4 {\n            },\n            Delta: 5 {\n            },\n            MapData: 6 {\n                // FIXME serialize/deserialize as `name(type)`\n                function: PgProcId,\n            },\n            MapSeries: 7 {\n                // FIXME serialize/deserialize as `name(type)`\n                function: PgProcId,\n            },\n            Arithmetic: 8 {\n                function: arithmetic::Function,\n                rhs: f64,\n            },\n            MapLambda: 9 {\n                lambda: LambdaData<'input>,\n            },\n            FilterLambda: 10 {\n                lambda: LambdaData<'input>,\n            },\n            FillTo: 11 {\n                interval: i64,\n                fill_method: FillToMethod,\n            },\n        }\n    }\n\n    impl<'input> Element<'input> {\n        pub fn flatten<'a>(self) -> UnstableTimevectorPipeline<'a> {\n            // TODO it'd be nice not to have to allocate a vector here but\n            //      `let slice = &[self][..];`\n            //      gives a lifetime error I don't yet know how to solve\n            let slice = vec![self].into();\n            unsafe {\n                flatten! {\n                    UnstableTimevectorPipeline {\n                        num_elements: 1,\n                        elements: slice,\n                    }\n                }\n            }\n        }\n    }\n\n    impl<'e> From<Element<'e>> for UnstableTimevectorPipeline<'e> {\n        fn from(element: Element<'e>) -> Self {\n            build! {\n                UnstableTimevectorPipeline {\n                    num_elements: 1,\n                    elements: vec![element].into(),\n                }\n            }\n        }\n    }\n\n    ron_inout_funcs!(UnstableTimevectorPipeline<'input>);\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_run_pipeline<'a>(\n    timevector: Timevector_TSTZ_F64<'a>,\n    pipeline: toolkit_experimental::UnstableTimevectorPipeline<'a>,\n) -> Timevector_TSTZ_F64<'static> {\n    run_pipeline_elements(timevector, pipeline.elements.iter()).in_current_context()\n}\n\npub fn run_pipeline_elements<'s, 'j, 'i>(\n    mut timevector: Timevector_TSTZ_F64<'s>,\n    pipeline: impl Iterator<Item = Element<'j>> + 'i,\n) -> Timevector_TSTZ_F64<'s> {\n    for element in pipeline {\n        timevector = execute_pipeline_element(timevector, &element);\n    }\n    timevector\n}\n\npub fn execute_pipeline_element<'s>(\n    timevector: Timevector_TSTZ_F64<'s>,\n    element: &Element,\n) -> Timevector_TSTZ_F64<'s> {\n    match element {\n        Element::LTTB { resolution } => crate::lttb::lttb_ts(timevector, *resolution as _),\n        Element::Sort { .. } => sort_timevector(timevector),\n        Element::Delta { .. } => timevector_delta(&timevector),\n        Element::MapData { function } => map::apply_to(timevector, function.0),\n        Element::MapSeries { function } => map::apply_to_series(timevector, function.0),\n        Element::MapLambda { lambda } => map::apply_lambda_to(timevector, lambda),\n        Element::FilterLambda { lambda } => filter::apply_lambda_to(timevector, lambda),\n        Element::Arithmetic { function, rhs } => arithmetic::apply(timevector, *function, *rhs),\n        Element::FillTo { .. } => fill_to(timevector, element),\n    }\n}\n\n// TODO is (immutable, parallel_safe) correct?\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_add_unstable_element<'p>(\n    mut pipeline: toolkit_experimental::UnstableTimevectorPipeline<'p>,\n    element: toolkit_experimental::UnstableTimevectorPipeline<'p>,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'p> {\n    pipeline.elements.as_owned().extend(element.elements.iter());\n    pipeline.num_elements = pipeline.elements.len().try_into().unwrap();\n    pipeline\n}\n\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    schema = \"toolkit_experimental\",\n    name = \"toolkit_pipeline_support\"\n)]\npub unsafe fn pipeline_support(input: Internal) -> Internal {\n    pipeline_support_helper(input, |old_pipeline, new_element| {\n        let new_element = UnstableTimevectorPipeline::from_polymorphic_datum(\n            new_element,\n            false,\n            pg_sys::Oid::INVALID,\n        )\n        .unwrap();\n        arrow_add_unstable_element(old_pipeline, new_element)\n            .into_datum()\n            .unwrap()\n    })\n}\n\npub(crate) unsafe fn pipeline_support_helper(\n    input: Internal,\n    make_new_pipeline: impl FnOnce(UnstableTimevectorPipeline, pg_sys::Datum) -> pg_sys::Datum,\n) -> Internal {\n    use std::mem::{size_of, MaybeUninit};\n\n    let input = input.unwrap().unwrap();\n    let input: *mut pg_sys::Node = input.cast_mut_ptr();\n    if !pgrx::is_a(input, pg_sys::NodeTag::T_SupportRequestSimplify) {\n        return no_change();\n    }\n\n    let req: *mut pg_sys::SupportRequestSimplify = input.cast();\n\n    let final_executor = (*req).fcall;\n    let original_args = PgList::from_pg((*final_executor).args);\n    assert_eq!(original_args.len(), 2);\n    let arg1 = original_args.head().unwrap();\n    let arg2 = original_args.tail().unwrap();\n\n    let (executor_id, lhs_args) = if is_a(arg1, pg_sys::NodeTag::T_OpExpr) {\n        let old_executor: *mut pg_sys::OpExpr = arg1.cast();\n        ((*old_executor).opfuncid, (*old_executor).args)\n    } else if is_a(arg1, pg_sys::NodeTag::T_FuncExpr) {\n        let old_executor: *mut pg_sys::FuncExpr = arg1.cast();\n        ((*old_executor).funcid, (*old_executor).args)\n    } else {\n        return no_change();\n    };\n\n    // check old_executor operator fn is 'run_pipeline' above\n    static RUN_PIPELINE_OID: once_cell::sync::OnceCell<pg_sys::Oid> =\n        once_cell::sync::OnceCell::new();\n    match RUN_PIPELINE_OID.get() {\n        Some(oid) => {\n            if executor_id != *oid {\n                return no_change();\n            }\n        }\n        None => {\n            let executor_fn = {\n                let mut flinfo: pg_sys::FmgrInfo = MaybeUninit::zeroed().assume_init();\n                pg_sys::fmgr_info(executor_id, &mut flinfo);\n                flinfo.fn_addr\n            };\n            // FIXME this cast should not be necessary; pgrx is defining the\n            //       wrapper functions as\n            //       `unsafe fn(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum`\n            //       instead of\n            //       `unsafe extern \"C\" fn(fcinfo: pg_sys::FunctionCallInfo) -> pg_sys::Datum`\n            //       we'll fix this upstream\n            let expected_executor = arrow_run_pipeline_wrapper as usize;\n            match executor_fn {\n                None => return no_change(),\n                // FIXME the direct comparison should work\n                Some(func) if func as usize != expected_executor => return no_change(),\n                Some(_) => RUN_PIPELINE_OID.get_or_init(|| executor_id),\n            };\n        }\n    }\n\n    let lhs_args = PgList::from_pg(lhs_args);\n    assert_eq!(lhs_args.len(), 2);\n    let old_series = lhs_args.head().unwrap();\n    let old_const = lhs_args.tail().unwrap();\n\n    if !is_a(old_const, pg_sys::NodeTag::T_Const) {\n        return no_change();\n    }\n\n    let old_const: *mut pg_sys::Const = old_const.cast();\n\n    if !is_a(arg2, pg_sys::NodeTag::T_Const) {\n        return no_change();\n    }\n\n    let new_element_const: *mut pg_sys::Const = arg2.cast();\n\n    let old_pipeline = UnstableTimevectorPipeline::from_polymorphic_datum(\n        (*old_const).constvalue,\n        false,\n        pg_sys::Oid::INVALID,\n    )\n    .unwrap();\n    let new_pipeline = make_new_pipeline(old_pipeline, (*new_element_const).constvalue);\n\n    let new_const = pg_sys::palloc(size_of::<pg_sys::Const>()).cast();\n    *new_const = *new_element_const;\n    (*new_const).constvalue = new_pipeline;\n\n    let new_executor = pg_sys::palloc(size_of::<pg_sys::FuncExpr>()).cast();\n    *new_executor = *final_executor;\n    let mut new_executor_args = PgList::new();\n    new_executor_args.push(old_series);\n    new_executor_args.push(new_const.cast());\n    (*new_executor).args = new_executor_args.into_pg();\n\n    Internal::from(Some(pg_sys::Datum::from(new_executor)))\n}\n\n// support functions are spec'd as returning NULL pointer if no simplification\n// can be made\nfn no_change() -> pgrx::Internal {\n    Internal::from(Some(pg_sys::Datum::from(\n        std::ptr::null_mut::<pg_sys::Expr>(),\n    )))\n}\n\n// using this instead of pg_operator since the latter doesn't support schemas yet\n// FIXME there is no CREATE OR REPLACE OPERATOR need to update post-install.rs\n//       need to ensure this works with out unstable warning\nextension_sql!(\n    r#\"\nALTER FUNCTION \"arrow_run_pipeline\" SUPPORT toolkit_experimental.toolkit_pipeline_support;\nALTER FUNCTION \"arrow_add_unstable_element\" SUPPORT toolkit_experimental.toolkit_pipeline_support;\n\"#,\n    name = \"pipe_support\",\n    requires = [pipeline_support],\n);\n\n// TODO is (immutable, parallel_safe) correct?\n#[pg_extern(\n    immutable,\n    parallel_safe,\n    name = \"lttb\",\n    schema = \"toolkit_experimental\"\n)]\npub fn lttb_pipeline_element(\n    resolution: i32,\n) -> toolkit_experimental::UnstableTimevectorPipeline<'static> {\n    Element::LTTB {\n        resolution: resolution.try_into().unwrap(),\n    }\n    .flatten()\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_pipeline_lttb() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE TABLE lttb_pipe (series timevector_tstz_f64)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\n                \"INSERT INTO lttb_pipe \\\n                SELECT timevector(time, val) FROM ( \\\n                    SELECT \\\n                        '2020-01-01 UTC'::TIMESTAMPTZ + make_interval(days=>(foo*10)::int) as time, \\\n                        TRUNC((10 + 5 * cos(foo))::numeric, 4) as val \\\n                    FROM generate_series(1,11,0.1) foo \\\n                ) bar\",\n                None,\n                &[]\n            ).unwrap();\n\n            let val = client\n                .update(\n                    \"SELECT (series -> lttb(17))::TEXT FROM lttb_pipe\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:17,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-11 00:00:00+00\\\",val:12.7015),\\\n                (ts:\\\"2020-01-13 00:00:00+00\\\",val:11.8117),\\\n                (ts:\\\"2020-01-22 00:00:00+00\\\",val:7.4757),\\\n                (ts:\\\"2020-01-28 00:00:00+00\\\",val:5.4796),\\\n                (ts:\\\"2020-02-03 00:00:00+00\\\",val:5.0626),\\\n                (ts:\\\"2020-02-09 00:00:00+00\\\",val:6.3703),\\\n                (ts:\\\"2020-02-14 00:00:00+00\\\",val:8.4633),\\\n                (ts:\\\"2020-02-24 00:00:00+00\\\",val:13.1734),\\\n                (ts:\\\"2020-03-01 00:00:00+00\\\",val:14.8008),\\\n                (ts:\\\"2020-03-07 00:00:00+00\\\",val:14.7511),\\\n                (ts:\\\"2020-03-13 00:00:00+00\\\",val:13.0417),\\\n                (ts:\\\"2020-03-23 00:00:00+00\\\",val:8.3042),\\\n                (ts:\\\"2020-03-29 00:00:00+00\\\",val:5.9445),\\\n                (ts:\\\"2020-04-04 00:00:00+00\\\",val:5.0015),\\\n                (ts:\\\"2020-04-10 00:00:00+00\\\",val:5.8046),\\\n                (ts:\\\"2020-04-14 00:00:00+00\\\",val:7.195),\\\n                (ts:\\\"2020-04-20 00:00:00+00\\\",val:10.0221)\\\n            ],null_val:[0,0,0])\"\n            );\n\n            let val = client\n                .update(\"SELECT (series -> lttb(8))::TEXT FROM lttb_pipe\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:8,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-11 00:00:00+00\\\",val:12.7015),\\\n                (ts:\\\"2020-01-27 00:00:00+00\\\",val:5.7155),\\\n                (ts:\\\"2020-02-06 00:00:00+00\\\",val:5.5162),\\\n                (ts:\\\"2020-02-27 00:00:00+00\\\",val:14.1735),\\\n                (ts:\\\"2020-03-09 00:00:00+00\\\",val:14.3469),\\\n                (ts:\\\"2020-03-30 00:00:00+00\\\",val:5.6728),\\\n                (ts:\\\"2020-04-09 00:00:00+00\\\",val:5.554),\\\n                (ts:\\\"2020-04-20 00:00:00+00\\\",val:10.0221)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    \"SELECT (series -> lttb(8) -> lttb(8))::TEXT FROM lttb_pipe\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:8,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-11 00:00:00+00\\\",val:12.7015),\\\n                (ts:\\\"2020-01-27 00:00:00+00\\\",val:5.7155),\\\n                (ts:\\\"2020-02-06 00:00:00+00\\\",val:5.5162),\\\n                (ts:\\\"2020-02-27 00:00:00+00\\\",val:14.1735),\\\n                (ts:\\\"2020-03-09 00:00:00+00\\\",val:14.3469),\\\n                (ts:\\\"2020-03-30 00:00:00+00\\\",val:5.6728),\\\n                (ts:\\\"2020-04-09 00:00:00+00\\\",val:5.554),\\\n                (ts:\\\"2020-04-20 00:00:00+00\\\",val:10.0221)\\\n            ],null_val:[0])\"\n            );\n\n            let val = client\n                .update(\n                    \"SELECT (series -> (lttb(8) -> lttb(8) -> lttb(8)))::TEXT FROM lttb_pipe\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(\n                val.unwrap(),\n                \"(version:1,num_points:8,flags:1,internal_padding:(0,0,0),points:[\\\n                (ts:\\\"2020-01-11 00:00:00+00\\\",val:12.7015),\\\n                (ts:\\\"2020-01-27 00:00:00+00\\\",val:5.7155),\\\n                (ts:\\\"2020-02-06 00:00:00+00\\\",val:5.5162),\\\n                (ts:\\\"2020-02-27 00:00:00+00\\\",val:14.1735),\\\n                (ts:\\\"2020-03-09 00:00:00+00\\\",val:14.3469),\\\n                (ts:\\\"2020-03-30 00:00:00+00\\\",val:5.6728),\\\n                (ts:\\\"2020-04-09 00:00:00+00\\\",val:5.554),\\\n                (ts:\\\"2020-04-20 00:00:00+00\\\",val:10.0221)\\\n            ],null_val:[0])\"\n            );\n        });\n    }\n\n    #[pg_test]\n    fn test_pipeline_folding() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            // using the search path trick for this test b/c the operator is\n            // difficult to spot otherwise.\n            let sp = client\n                .update(\n                    \"SELECT format(' %s, toolkit_experimental',current_setting('search_path'))\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            client\n                .update(&format!(\"SET LOCAL search_path TO {sp}\"), None, &[])\n                .unwrap();\n\n            let output = client.update(\n                \"EXPLAIN (verbose) SELECT timevector('2021-01-01'::timestamptz, 0.1) -> round() -> abs() -> round();\",\n                None,\n                &[]\n            ).unwrap().nth(1)\n                .unwrap()\n                .get_datum_by_ordinal(1).unwrap()\n                .value::<String>().unwrap().unwrap();\n            // check that it's executing as if we had input `timevector -> (round() -> abs())`\n            assert_eq!(output.trim(), \"Output: \\\n                arrow_run_pipeline(\\\n                    timevector('2021-01-01 00:00:00+00'::timestamp with time zone, '0.1'::double precision), \\\n                   '(version:1,num_elements:3,elements:[\\\n                        Arithmetic(function:Round,rhs:0),\\\n                        Arithmetic(function:Abs,rhs:0),\\\n                        Arithmetic(function:Round,rhs:0)\\\n                    ])'::unstabletimevectorpipeline\\\n                )\");\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/time_vector.rs",
    "content": "#![allow(clippy::identity_op)] // clippy gets confused by pg_type! enums\n\nuse crate::pg_sys::timestamptz_to_str;\nuse core::str::Utf8Error;\nuse pgrx::{iter::TableIterator, *};\nuse std::ffi::CStr;\nuse tera::{Context, Tera};\n\nuse crate::{\n    aggregate_utils::in_aggregate_context,\n    build, flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\nuse tspoint::TSPoint;\n\npub use iter::Iter;\n\nuse flat_serialize::*;\n\nmod iter;\nmod pipeline;\n\nuse crate::raw::bytea;\n\n// Bit flags stored in Timevector flags\npub const FLAG_IS_SORTED: u8 = 0x01;\npub const FLAG_HAS_NULLS: u8 = 0x01 << 1;\n\npg_type! {\n    #[derive(Debug)]\n    #[allow(non_camel_case_types)]\n    struct Timevector_TSTZ_F64<'input> {\n        num_points: u32,\n        flags: u8,         // extra information about the stored data\n        internal_padding: [u8; 3],  // required to be aligned\n        points: [TSPoint; self.num_points],\n        null_val: [u8; self.num_points.div_ceil(8)], // bit vector, must be last element for alignment purposes\n    }\n}\n\nron_inout_funcs!(Timevector_TSTZ_F64<'input>);\n\nimpl<'input> Timevector_TSTZ_F64<'input> {\n    pub fn num_points(&self) -> usize {\n        self.num_points as usize\n    }\n\n    // Gets the nth point of a timevector\n    // Differs from normal vector get in that it returns a copy rather than a reference (as the point may have to be constructed)\n    pub fn get(&self, index: usize) -> Option<TSPoint> {\n        if index >= self.num_points() {\n            return None;\n        }\n\n        Some(self.points.as_slice()[index])\n    }\n\n    #[inline]\n    pub fn is_sorted(&self) -> bool {\n        self.flags & FLAG_IS_SORTED != 0\n    }\n\n    #[inline]\n    pub fn has_nulls(&self) -> bool {\n        self.flags & FLAG_HAS_NULLS != 0\n    }\n\n    pub fn is_null_val(&self, index: usize) -> bool {\n        assert!(index < self.num_points()); // should we handle this better\n\n        let byte_id = index / 8;\n        let byte_idx = index % 8;\n\n        self.null_val.as_slice()[byte_id] & (1 << byte_idx) != 0\n    }\n\n    fn clone_owned(&self) -> Timevector_TSTZ_F64<'static> {\n        Timevector_TSTZ_F64Data::clone(self).into_owned().into()\n    }\n}\n\nimpl<'a> Timevector_TSTZ_F64<'a> {\n    pub fn iter(&self) -> Iter<'_> {\n        Iter::Slice {\n            iter: self.points.iter(),\n        }\n    }\n\n    pub fn num_vals(&self) -> usize {\n        self.num_points()\n    }\n}\n\nimpl<'a> IntoIterator for Timevector_TSTZ_F64<'a> {\n    type Item = TSPoint;\n    type IntoIter = Iter<'a>;\n\n    fn into_iter(self) -> Self::IntoIter {\n        #[allow(clippy::unnecessary_to_owned)] // Pretty sure clippy's wrong about this\n        Iter::Slice {\n            iter: self.points.to_owned().into_iter(),\n        }\n    }\n}\n\npub static TIMEVECTOR_OID: once_cell::sync::Lazy<pg_sys::Oid> =\n    once_cell::sync::Lazy::new(Timevector_TSTZ_F64::type_oid);\n\n#[pg_extern(immutable, parallel_safe)]\npub fn unnest<'a>(\n    series: Timevector_TSTZ_F64<'a>,\n) -> TableIterator<'a, (name!(time, crate::raw::TimestampTz), name!(value, f64))> {\n    TableIterator::new(\n        series\n            .into_iter()\n            .map(|points| (points.ts.into(), points.val)),\n    )\n}\n\n/// Util function to convert from *const ::std::os::raw::c_char to String\n/// TimestampTz -> *const c_char -> &CStr -> &str -> String\npub fn timestamptz_to_string(time: pg_sys::TimestampTz) -> Result<String, Utf8Error> {\n    let char_ptr = unsafe { timestamptz_to_str(time) };\n    let c_str = unsafe { CStr::from_ptr(char_ptr) };\n    c_str.to_str().map(|s| s.to_owned())\n}\n\n#[pg_extern(immutable, schema = \"toolkit_experimental\", parallel_safe)]\npub fn to_plotly<'a>(series: Timevector_TSTZ_F64<'a>) -> String {\n    format_timevector(series,\"{\\\"times\\\": {{ TIMES | json_encode() | safe  }}, \\\"vals\\\": {{ VALUES | json_encode() | safe }}}\".to_string())\n}\n\n#[pg_extern(immutable, schema = \"toolkit_experimental\", parallel_safe)]\npub fn to_text<'a>(series: Timevector_TSTZ_F64<'a>, format_string: String) -> String {\n    format_timevector(series, format_string)\n}\n\npub fn format_timevector<'a>(series: Timevector_TSTZ_F64<'a>, format_string: String) -> String {\n    let mut context = Context::new();\n    let mut times: Vec<String> = Vec::new();\n    let mut values: Vec<String> = Vec::new();\n    if series.has_nulls() {\n        for (i, point) in series.iter().enumerate() {\n            times.push(timestamptz_to_string(point.ts).unwrap());\n            if series.is_null_val(i) {\n                values.push(\"null\".to_string())\n            } else {\n                match point.val.to_string().as_ref() {\n                    \"NaN\" | \"inf\" | \"-inf\" | \"Infinity\" | \"-Infinity\" => {\n                        panic!(\"All values in the series must be finite\")\n                    }\n                    x => values.push(x.to_string()),\n                }\n            }\n        }\n    } else {\n        // optimized path if series does not have any nulls, but might have some NaNs/infinities\n        for point in series {\n            times.push(timestamptz_to_string(point.ts).unwrap());\n            match point.val.to_string().as_ref() {\n                \"NaN\" | \"inf\" | \"-inf\" | \"Infinity\" | \"-Infinity\" => {\n                    panic!(\"All values in the series must be finite\")\n                }\n                x => values.push(x.to_string()),\n            }\n        }\n    }\n\n    context.insert(\"TIMES\", &times);\n    context.insert(\"VALUES\", &values);\n\n    // paired timevals in the following format: [{\\\"time\\\": \\\"2020-01-01 00:00:00+00\\\", \\\"val\\\": 1}, {\\\"time\\\": \\\"2020-01-02 00:00:00+00\\\", \\\"val\\\": 2}, ... ]\n    let timevals = Tera::one_off(\"[{% for x in TIMES %}{\\\"time\\\": \\\"{{ x }}\\\", \\\"val\\\": {{ VALUES[loop.index0] }}}{% if not loop.last %},{% endif %} {% endfor %}]\", &context,false).expect(\"Failed to create paired template\");\n    context.insert(\"TIMEVALS\", &timevals);\n    Tera::one_off(format_string.as_ref(), &context, false)\n        .expect(\"Failed to create template with Tera\")\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_timevector_unnest<'a>(\n    series: Timevector_TSTZ_F64<'a>,\n    _accessor: crate::accessors::AccessorUnnest,\n) -> TableIterator<'a, (name!(time, crate::raw::TimestampTz), name!(value, f64))> {\n    unnest(series)\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn timevector_serialize(state: Internal) -> bytea {\n    let state: &Timevector_TSTZ_F64 = unsafe { state.get().unwrap() };\n    let state: &Timevector_TSTZ_F64Data = &state.0;\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn timevector_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    let data: Timevector_TSTZ_F64<'static> = crate::do_deserialize!(bytes, Timevector_TSTZ_F64Data);\n    Inner::from(data).internal()\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn timevector_tstz_f64_trans(\n    state: Internal,\n    time: Option<crate::raw::TimestampTz>,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { timevector_trans_inner(state.to_inner(), time, value, fcinfo).internal() }\n}\n\npub fn timevector_trans_inner(\n    state: Option<Inner<Timevector_TSTZ_F64<'_>>>,\n    time: Option<crate::raw::TimestampTz>,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<Timevector_TSTZ_F64<'_>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let time: pg_sys::TimestampTz = match time {\n                None => return state,\n                Some(time) => time.into(),\n            };\n            let mut state = match state {\n                None => Inner::from(build! {\n                    Timevector_TSTZ_F64 {\n                        num_points: 0,\n                        flags: FLAG_IS_SORTED,\n                        internal_padding: [0; 3],\n                        points: vec![].into(),\n                        null_val: vec![].into(),\n                    }\n                }),\n                Some(state) => state,\n            };\n            if let Some(last_point) = state.points.as_slice().last() {\n                if state.is_sorted() && last_point.ts > time {\n                    state.flags ^= FLAG_IS_SORTED;\n                }\n            }\n            if state.num_points % 8 == 0 {\n                state.null_val.as_owned().push(0);\n            }\n            match value {\n                None => {\n                    state.flags |= FLAG_HAS_NULLS;\n                    state.points.as_owned().push(TSPoint {\n                        ts: time,\n                        val: f64::NAN,\n                    });\n                    let byte_idx = state.num_points % 8; // off by 1, but num_points isn't yet incremented\n                    *state.null_val.as_owned().last_mut().unwrap() |= 1 << byte_idx;\n                }\n                Some(val) => state.points.as_owned().push(TSPoint { ts: time, val }),\n            };\n            state.num_points += 1;\n            Some(state)\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn timevector_tstz_f64_compound_trans<'a>(\n    state: Internal,\n    series: Option<Timevector_TSTZ_F64<'a>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    inner_compound_trans(unsafe { state.to_inner() }, series, fcinfo).internal()\n}\n\npub fn inner_compound_trans<'b>(\n    state: Option<Inner<Timevector_TSTZ_F64<'static>>>,\n    series: Option<Timevector_TSTZ_F64<'b>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<Timevector_TSTZ_F64<'static>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, series) {\n            (None, None) => None,\n            (Some(state), None) => Some(state),\n            (None, Some(series)) => Some(series.clone_owned().into()),\n            (Some(state), Some(series)) => {\n                // TODO: this should be doable without cloning 'state'\n                Some(combine(state.clone(), series.clone()).into())\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn timevector_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { inner_combine(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\n\npub fn inner_combine<'a, 'b>(\n    state1: Option<Inner<Timevector_TSTZ_F64<'a>>>,\n    state2: Option<Inner<Timevector_TSTZ_F64<'b>>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<Timevector_TSTZ_F64<'static>>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(state2)) => Some(state2.clone_owned().into()),\n            (Some(state1), None) => Some(state1.clone_owned().into()),\n            (Some(state1), Some(state2)) => Some(combine(state1.clone(), state2.clone()).into()),\n        })\n    }\n}\n\npub fn combine(\n    first: Timevector_TSTZ_F64<'_>,\n    second: Timevector_TSTZ_F64<'_>,\n) -> Timevector_TSTZ_F64<'static> {\n    if first.num_vals() == 0 {\n        return second.clone_owned();\n    }\n    if second.num_vals() == 0 {\n        return first.clone_owned();\n    }\n\n    let is_sorted = first.is_sorted()\n        && second.is_sorted()\n        && first.points.as_slice().last().unwrap().ts\n            <= second.points.as_slice().first().unwrap().ts;\n    let points: Vec<_> = first.iter().chain(second.iter()).collect();\n\n    let mut flags = (first.flags & FLAG_HAS_NULLS) | (second.flags & FLAG_HAS_NULLS);\n    if is_sorted {\n        flags |= FLAG_IS_SORTED;\n    }\n\n    let null_val = if flags & FLAG_HAS_NULLS == 0 {\n        std::vec::from_elem(0_u8, points.len().div_ceil(8))\n    } else {\n        let mut v = first.null_val.as_slice().to_vec();\n        v.resize(points.len().div_ceil(8), 0);\n        if second.has_nulls() {\n            for i in 0..second.num_points {\n                if second.is_null_val(i as usize) {\n                    let idx = i + first.num_points;\n                    let byte_id = idx / 8;\n                    let byte_idx = idx % 8;\n                    v[byte_id as usize] |= 1 << byte_idx;\n                }\n            }\n        }\n        v\n    };\n\n    build! {\n        Timevector_TSTZ_F64 {\n            num_points: points.len() as _,\n            flags,\n            internal_padding: [0; 3],\n            points: points.into(),\n            null_val: null_val.into(),\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn timevector_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    unsafe { timevector_final_inner(state.to_inner(), fcinfo) }\n}\n\npub fn timevector_final_inner<'a>(\n    state: Option<Inner<Timevector_TSTZ_F64<'a>>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Timevector_TSTZ_F64<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let state = match state {\n                None => return None,\n                Some(state) => state,\n            };\n            Some(state.in_current_context())\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE timevector(ts TIMESTAMPTZ, value DOUBLE PRECISION) (\\n\\\n        sfunc = timevector_tstz_f64_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = timevector_final,\\n\\\n        combinefunc = timevector_combine,\\n\\\n        serialfunc = timevector_serialize,\\n\\\n        deserialfunc = timevector_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"timevector_tstz_f64_agg\",\n    requires = [\n        timevector_tstz_f64_trans,\n        timevector_final,\n        timevector_combine,\n        timevector_serialize,\n        timevector_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\nCREATE AGGREGATE rollup(\\n\\\n    timevector_tstz_f64\\n\\\n) (\\n\\\n    sfunc = timevector_tstz_f64_compound_trans,\\n\\\n    stype = internal,\\n\\\n    finalfunc = timevector_final,\\n\\\n    combinefunc = timevector_combine,\\n\\\n    serialfunc = timevector_serialize,\\n\\\n    deserialfunc = timevector_deserialize,\\n\\\n    parallel = safe\\n\\\n);\\n\\\n\",\n    name = \"timevector_tstz_f64_rollup\",\n    requires = [\n        timevector_tstz_f64_compound_trans,\n        timevector_final,\n        timevector_combine,\n        timevector_serialize,\n        timevector_deserialize\n    ],\n);\n\n#[pg_schema]\npub mod toolkit_experimental {\n    use super::*;\n\n    // Only making this available through the arrow operator right now, as the semantics are cleaner that way\n    pub fn asof_join<'a, 'b>(\n        from: Timevector_TSTZ_F64<'a>,\n        into: Timevector_TSTZ_F64<'b>,\n    ) -> TableIterator<\n        'a,\n        (\n            name!(value1, Option<f64>),\n            name!(value2, f64),\n            name!(time, crate::raw::TimestampTz),\n        ),\n    > {\n        assert!(\n            from.num_points > 0 && into.num_points > 0,\n            \"both timevectors must be populated for an asof join\"\n        );\n        let mut from = from\n            .into_iter()\n            .map(|points| (points.ts.into(), points.val))\n            .peekable();\n        let into = into.into_iter().map(|points| (points.ts, points.val));\n        let (mut from_time, mut from_val) = from.next().unwrap();\n\n        let mut results = vec![];\n        for (into_time, into_val) in into {\n            // Handle case where into starts before from\n            if into_time < from_time {\n                results.push((None, into_val, crate::raw::TimestampTz::from(into_time)));\n                continue;\n            }\n\n            while let Some((peek_time, _)) = from.peek() {\n                if *peek_time > into_time {\n                    break;\n                }\n                (from_time, from_val) = from.next().unwrap();\n            }\n\n            results.push((\n                Some(from_val),\n                into_val,\n                crate::raw::TimestampTz::from(into_time),\n            ));\n        }\n\n        TableIterator::new(results.into_iter())\n    }\n\n    pg_type! {\n        #[derive(Debug)]\n        struct AccessorAsof<'input> {\n            into: Timevector_TSTZ_F64Data<'input>,\n        }\n    }\n\n    ron_inout_funcs!(AccessorAsof<'input>);\n\n    #[pg_extern(immutable, parallel_safe, name = \"asof\")]\n    pub fn accessor_asof<'a>(tv: Timevector_TSTZ_F64<'a>) -> AccessorAsof<'static> {\n        unsafe {\n            flatten! {\n                AccessorAsof {\n                    into: tv.0\n                }\n            }\n        }\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_timevector_asof<'a>(\n    series: Timevector_TSTZ_F64<'a>,\n    accessor: toolkit_experimental::AccessorAsof,\n) -> TableIterator<\n    'a,\n    (\n        name!(value1, Option<f64>),\n        name!(value2, f64),\n        name!(time, crate::raw::TimestampTz),\n    ),\n> {\n    toolkit_experimental::asof_join(series, accessor.into.clone().into())\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    pub fn test_unnest() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO data VALUES\n                    ('2020-1-1', 30.0),\n                    ('2020-1-2', 45.0),\n                    ('2020-1-3', NULL),\n                    ('2020-1-4', 55.5),\n                    ('2020-1-5', 10.0)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut unnest = client\n                .update(\n                    \"SELECT unnest(timevector(time, value))::TEXT FROM data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",30)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-02 00:00:00+00\\\",45)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-03 00:00:00+00\\\",NaN)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",55.5)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",10)\")\n            );\n            assert!(unnest.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    pub fn test_format_timevector() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO data VALUES\n                    ('2020-1-1', 30.0),\n                    ('2020-1-2', 45.0),\n                    ('2020-1-3', NULL),\n                    ('2020-1-4', 55.5),\n                    ('2020-1-5', 10.0)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let test_plotly_template = client\n                .update(\n                    \"SELECT toolkit_experimental.to_plotly(timevector(time, value)) FROM data\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n\n            assert_eq!(test_plotly_template,\n\"{\\\"times\\\": [\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-02 00:00:00+00\\\",\\\"2020-01-03 00:00:00+00\\\",\\\"2020-01-04 00:00:00+00\\\",\\\"2020-01-05 00:00:00+00\\\"], \\\"vals\\\": [\\\"30\\\",\\\"45\\\",\\\"null\\\",\\\"55.5\\\",\\\"10\\\"]}\"\n\t\t     );\n            let test_paired_timevals_template = client.update(\n                \"SELECT toolkit_experimental.to_text(timevector(time, value),'{{TIMEVALS}}') FROM data\",\n                None,\n                &[]\n            ).unwrap().first()\n                .get_one::<String>().unwrap()\n                .unwrap();\n\n            assert_eq!(\n                test_paired_timevals_template,\"[{\\\"time\\\": \\\"2020-01-01 00:00:00+00\\\", \\\"val\\\": 30}, {\\\"time\\\": \\\"2020-01-02 00:00:00+00\\\", \\\"val\\\": 45}, {\\\"time\\\": \\\"2020-01-03 00:00:00+00\\\", \\\"val\\\": null}, {\\\"time\\\": \\\"2020-01-04 00:00:00+00\\\", \\\"val\\\": 55.5}, {\\\"time\\\": \\\"2020-01-05 00:00:00+00\\\", \\\"val\\\": 10} ]\"\n            );\n\n            let test_user_supplied_template = client\n                .update(\n                    \"SELECT toolkit_experimental.to_text(timevector(time,value), '{\\\"times\\\": {{ TIMES }}, \\\"vals\\\": {{ VALUES }}}') FROM data\",\n                    None,\n                    &[]\n                )\n                .unwrap().first()\n                .get_one::<String>().unwrap()\n                .unwrap();\n            assert_eq!(\n                test_user_supplied_template,\"{\\\"times\\\": [2020-01-01 00:00:00+00, 2020-01-02 00:00:00+00, 2020-01-03 00:00:00+00, 2020-01-04 00:00:00+00, 2020-01-05 00:00:00+00], \\\"vals\\\": [30, 45, null, 55.5, 10]}\"\n            );\n            let test_user_supplied_json_template = client.update(\n                \"SELECT toolkit_experimental.to_text(timevector(time, value),'{\\\"times\\\": {{ TIMES | json_encode() | safe  }}, \\\"vals\\\": {{ VALUES | json_encode() | safe }}}') FROM data\",\n                None,\n                &[]\n            ).unwrap().first()\n                .get_one::<String>().unwrap()\n                .unwrap();\n\n            assert_eq!(\n                test_user_supplied_json_template,\n\"{\\\"times\\\": [\\\"2020-01-01 00:00:00+00\\\",\\\"2020-01-02 00:00:00+00\\\",\\\"2020-01-03 00:00:00+00\\\",\\\"2020-01-04 00:00:00+00\\\",\\\"2020-01-05 00:00:00+00\\\"], \\\"vals\\\": [\\\"30\\\",\\\"45\\\",\\\"null\\\",\\\"55.5\\\",\\\"10\\\"]}\"\n            );\n        })\n    }\n\n    #[should_panic = \"All values in the series must be finite\"]\n    #[pg_test]\n    pub fn test_format_timevector_panics_on_infinities() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO data VALUES\n                    ('2020-1-1', 30.0),\n                    ('2020-1-2', 45.0),\n                    ('2020-1-3', NULL),\n                    ('2020-1-4', 55.5),\n                    ('2020-1-6', 'Infinity'),\n                    ('2020-1-5', 10.0)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let test_plotly_template = client\n                .update(\n                    \"SELECT toolkit_experimental.to_plotly(timevector(time, value)) FROM data\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n\n            assert_eq!(test_plotly_template,\"{\\\"times\\\": [\\n  \\\"2020-01-01 00:00:00+00\\\",\\n  \\\"2020-01-02 00:00:00+00\\\",\\n  \\\"2020-01-03 00:00:00+00\\\",\\n  \\\"2020-01-04 00:00:00+00\\\",\\n  \\\"2020-01-05 00:00:00+00\\\"\\n], \\\"vals\\\": [\\n  \\\"30\\\",\\n  \\\"45\\\",\\n  \\\"null\\\",\\n  \\\"55.5\\\",\\n  \\\"10\\\"\\n]}\"\n\t\t     );\n        })\n    }\n\n    #[pg_test]\n    pub fn timevector_io() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO data VALUES\n                    ('2020-1-1', 30.0),\n                    ('2020-1-2', 45.0),\n                    ('2020-1-3', NULL),\n                    ('2020-1-4', 55.5),\n                    ('2020-1-5', 10.0)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let tvec = client\n                .update(\"SELECT timevector(time,value)::TEXT FROM data\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            let expected = r#\"(version:1,num_points:5,flags:3,internal_padding:(0,0,0),points:[(ts:\"2020-01-01 00:00:00+00\",val:30),(ts:\"2020-01-02 00:00:00+00\",val:45),(ts:\"2020-01-03 00:00:00+00\",val:NaN),(ts:\"2020-01-04 00:00:00+00\",val:55.5),(ts:\"2020-01-05 00:00:00+00\",val:10)],null_val:[4])\"#;\n\n            assert_eq!(tvec, expected);\n\n            let mut unnest = client\n                .update(\n                    &format!(\"SELECT unnest('{expected}'::timevector_tstz_f64)::TEXT\"),\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",30)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-02 00:00:00+00\\\",45)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-03 00:00:00+00\\\",NaN)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",55.5)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",10)\")\n            );\n            assert!(unnest.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    pub fn test_arrow_equivalence() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO data VALUES\n                    ('1-1-2020', 30.0),\n                    ('1-2-2020', 45.0),\n                    ('1-3-2020', NULL),\n                    ('1-4-2020', 55.5),\n                    ('1-5-2020', 10.0)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut func = client\n                .update(\n                    \"SELECT unnest(timevector(time, value))::TEXT FROM data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            let mut op = client\n                .update(\n                    \"SELECT (timevector(time, value) -> unnest())::TEXT FROM data\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut test = true;\n            while test {\n                match (func.next(), op.next()) {\n                    (None, None) => test = false,\n                    (Some(a), Some(b)) =>\n                        assert_eq!(a[1].value::<&str>(), b[1].value::<&str>()),\n                    _ => panic!(\"Arrow operator didn't contain the same number of elements as nested function\"),\n                };\n            }\n        })\n    }\n\n    #[pg_test]\n    pub fn test_rollup() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\n                    \"CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION, bucket INTEGER)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO data VALUES\n                    ('2020-1-1', 30.0, 1),\n                    ('2020-1-2', 45.0, 1),\n                    ('2020-1-3', NULL, 2),\n                    ('2020-1-4', 55.5, 2),\n                    ('2020-1-5', 10.0, 3),\n                    ('2020-1-6', 13.0, 3),\n                    ('2020-1-7', 71.0, 4),\n                    ('2020-1-8', 0.0, 4)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut unnest = client\n                .update(\n                    \"SELECT unnest(rollup(tvec))::TEXT\n                        FROM (\n                            SELECT timevector(time, value) AS tvec\n                            FROM data \n                            GROUP BY bucket \n                            ORDER BY bucket\n                        ) s\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-01 00:00:00+00\\\",30)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-02 00:00:00+00\\\",45)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-03 00:00:00+00\\\",NaN)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-04 00:00:00+00\\\",55.5)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-05 00:00:00+00\\\",10)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-06 00:00:00+00\\\",13)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-07 00:00:00+00\\\",71)\")\n            );\n            assert_eq!(\n                unnest.next().unwrap()[1].value().unwrap(),\n                Some(\"(\\\"2020-01-08 00:00:00+00\\\",0)\")\n            );\n            assert!(unnest.next().is_none());\n        })\n    }\n\n    #[pg_test]\n    fn test_rollup_preserves_nulls_flag() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            client\n                .update(\"CREATE TABLE tvecs (vector Timevector_TSTZ_F64)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO tvecs SELECT timevector('2020-1-1', 20)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO tvecs SELECT timevector('2020-1-2', 30)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO tvecs SELECT timevector('2020-1-3', 15)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let tvec = client\n                .update(\"SELECT rollup(vector)::TEXT FROM tvecs\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            let expected = r#\"(version:1,num_points:3,flags:1,internal_padding:(0,0,0),points:[(ts:\"2020-01-01 00:00:00+00\",val:20),(ts:\"2020-01-02 00:00:00+00\",val:30),(ts:\"2020-01-03 00:00:00+00\",val:15)],null_val:[0])\"#;\n            assert_eq!(tvec, expected);\n\n            client\n                .update(\n                    \"INSERT INTO tvecs SELECT timevector('2019-1-4', NULL)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            let tvec = client\n                .update(\"SELECT rollup(vector)::TEXT FROM tvecs\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap()\n                .unwrap();\n            let expected = r#\"(version:1,num_points:4,flags:2,internal_padding:(0,0,0),points:[(ts:\"2020-01-01 00:00:00+00\",val:20),(ts:\"2020-01-02 00:00:00+00\",val:30),(ts:\"2020-01-03 00:00:00+00\",val:15),(ts:\"2019-01-04 00:00:00+00\",val:NaN)],null_val:[8])\"#;\n            assert_eq!(tvec, expected);\n        })\n    }\n\n    #[pg_test]\n    fn test_asof_join() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            let mut result = client\n                .update(\n                    \"WITH s as (\n                    SELECT timevector(time, value) AS v1 FROM\n                    (VALUES \n                        ('2022-10-1 1:00 UTC'::TIMESTAMPTZ, 20.0),\n                        ('2022-10-1 2:00 UTC'::TIMESTAMPTZ, 30.0),\n                        ('2022-10-1 3:00 UTC'::TIMESTAMPTZ, 40.0)\n                    ) as v(time, value)),\n                t as (\n                    SELECT timevector(time, value) AS v2 FROM\n                    (VALUES \n                        ('2022-10-1 0:30 UTC'::TIMESTAMPTZ, 15.0),\n                        ('2022-10-1 2:00 UTC'::TIMESTAMPTZ, 45.0),\n                        ('2022-10-1 3:30 UTC'::TIMESTAMPTZ, 60.0)\n                    ) as v(time, value))\n                SELECT (v1 -> toolkit_experimental.asof(v2))::TEXT\n                FROM s, t;\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(,15,\\\"2022-10-01 00:30:00+00\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(30,45,\\\"2022-10-01 02:00:00+00\\\")\")\n            );\n            assert_eq!(\n                result.next().unwrap()[1].value().unwrap(),\n                Some(\"(40,60,\\\"2022-10-01 03:30:00+00\\\")\")\n            );\n            assert!(result.next().is_none());\n        })\n    }\n\n    #[pg_test(error = \"both timevectors must be populated for an asof join\")]\n    fn test_asof_none() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            client.update(\n                \"WITH s as (\n                    SELECT timevector(now(), 0) -> toolkit_experimental.filter($$ $value != 0 $$) AS empty),\n                    t as (\n                        SELECT timevector(time, value) AS valid FROM\n                        (VALUES \n                            ('2022-10-1 0:30 UTC'::TIMESTAMPTZ, 15.0),\n                            ('2022-10-1 2:00 UTC'::TIMESTAMPTZ, 45.0),\n                            ('2022-10-1 3:30 UTC'::TIMESTAMPTZ, 60.0)\n                        ) as v(time, value))\n                    SELECT (valid -> toolkit_experimental.asof(empty))\n                    FROM s, t;\", None, &[]).unwrap();\n        })\n    }\n\n    #[pg_test(error = \"both timevectors must be populated for an asof join\")]\n    fn test_none_asof() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n\n            client.update(\n                \"WITH s as (\n                    SELECT timevector(now(), 0) -> toolkit_experimental.filter($$ $value != 0 $$) AS empty),\n                    t as (\n                        SELECT timevector(time, value) AS valid FROM\n                        (VALUES \n                            ('2022-10-1 0:30 UTC'::TIMESTAMPTZ, 15.0),\n                            ('2022-10-1 2:00 UTC'::TIMESTAMPTZ, 45.0),\n                            ('2022-10-1 3:30 UTC'::TIMESTAMPTZ, 60.0)\n                        ) as v(time, value))\n                    SELECT (empty -> toolkit_experimental.asof(valid))\n                    FROM s, t;\", None, &[]).unwrap();\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/time_weighted_average/accessors.rs",
    "content": "use pgrx::*;\n\nuse crate::time_weighted_average::DurationUnit;\nuse crate::{\n    datum_utils::interval_to_ms,\n    flatten, pg_type, ron_inout_funcs,\n    time_weighted_average::{TimeWeightMethod, TimeWeightSummary, TimeWeightSummaryData},\n};\n\nuse tspoint::TSPoint;\n\npg_type! {\n    struct TimeWeightInterpolatedAverageAccessor {\n        timestamp : i64,\n        interval : i64,\n        prev : TimeWeightSummaryData,\n        pad : [u8;3],\n        flags : u32,\n        next : TimeWeightSummaryData,\n    }\n}\n\nron_inout_funcs!(TimeWeightInterpolatedAverageAccessor);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_average\")]\nfn time_weight_interpolated_average_accessor(\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: default!(Option<TimeWeightSummary>, \"NULL\"),\n    next: default!(Option<TimeWeightSummary>, \"NULL\"),\n) -> TimeWeightInterpolatedAverageAccessor {\n    fn empty_summary() -> Option<TimeWeightSummary> {\n        Some(unsafe {\n            flatten!(TimeWeightSummary {\n                first: TSPoint { ts: 0, val: 0.0 },\n                last: TSPoint { ts: 0, val: 0.0 },\n                weighted_sum: 0.0,\n                method: TimeWeightMethod::LOCF,\n            })\n        })\n    }\n\n    let flags = u32::from(prev.is_some()) + if next.is_some() { 2 } else { 0 };\n    let prev = prev.or_else(empty_summary).unwrap().0;\n    let next = next.or_else(empty_summary).unwrap().0;\n    let interval = interval_to_ms(&start, &duration);\n    crate::build! {\n        TimeWeightInterpolatedAverageAccessor {\n            timestamp : start.into(),\n            interval,\n            prev,\n            pad : [0,0,0],\n            flags,\n            next,\n        }\n    }\n}\n\npg_type! {\n    #[derive(Debug)]\n    struct TimeWeightInterpolatedIntegralAccessor {\n        start : i64,\n        interval : i64,\n        prev : TimeWeightSummaryData,\n        pad : [u8;3],\n        unit : u32,\n        flags: u64,\n        next : TimeWeightSummaryData,\n    }\n}\n\nron_inout_funcs!(TimeWeightInterpolatedIntegralAccessor);\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_integral\")]\nfn time_weight_interpolated_integral_accessor(\n    start: crate::raw::TimestampTz,\n    interval: crate::raw::Interval,\n    prev: default!(Option<TimeWeightSummary>, \"NULL\"),\n    next: default!(Option<TimeWeightSummary>, \"NULL\"),\n    unit: default!(String, \"'second'\"),\n) -> TimeWeightInterpolatedIntegralAccessor {\n    fn empty_summary() -> Option<TimeWeightSummary> {\n        Some(unsafe {\n            flatten!(TimeWeightSummary {\n                first: TSPoint { ts: 0, val: 0.0 },\n                last: TSPoint { ts: 0, val: 0.0 },\n                weighted_sum: 0.0,\n                method: TimeWeightMethod::LOCF,\n            })\n        })\n    }\n\n    let unit = match DurationUnit::from_str(&unit) {\n        Some(unit) => unit.microseconds(),\n        None => pgrx::error!(\n            \"Unrecognized duration unit: {}. Valid units are: usecond, msecond, second, minute, hour\",\n            unit,\n        ),\n    };\n    let flags = u64::from(prev.is_some()) + if next.is_some() { 2 } else { 0 };\n    let prev = prev.or_else(empty_summary).unwrap().0;\n    let next = next.or_else(empty_summary).unwrap().0;\n    let interval = interval_to_ms(&start, &interval);\n    crate::build! {\n        TimeWeightInterpolatedIntegralAccessor {\n            start: start.into(),\n            interval,\n            prev,\n            pad : [0,0,0],\n            unit,\n            flags,\n            next,\n        }\n    }\n}\n"
  },
  {
    "path": "extension/src/time_weighted_average.rs",
    "content": "#![allow(non_camel_case_types)]\n\nuse pgrx::*;\nuse serde::{Deserialize, Serialize};\n\nuse crate::{\n    accessors::{\n        AccessorAverage, AccessorFirstTime, AccessorFirstVal, AccessorIntegral, AccessorLastTime,\n        AccessorLastVal,\n    },\n    aggregate_utils::in_aggregate_context,\n    duration::DurationUnit,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type, ron_inout_funcs,\n};\n\nuse tspoint::TSPoint;\n\nuse time_weighted_average::{\n    TimeWeightError, TimeWeightMethod, TimeWeightSummary as TimeWeightSummaryInternal,\n};\n\nuse crate::raw::bytea;\n\nmod accessors;\n\nuse accessors::{TimeWeightInterpolatedAverageAccessor, TimeWeightInterpolatedIntegralAccessor};\n\npg_type! {\n    #[derive(Debug)]\n    struct TimeWeightSummary {\n        first: TSPoint,\n        last: TSPoint,\n        weighted_sum: f64,\n        method: TimeWeightMethod,\n    }\n}\nron_inout_funcs!(TimeWeightSummary);\n\nimpl TimeWeightSummary {\n    fn internal(&self) -> TimeWeightSummaryInternal {\n        TimeWeightSummaryInternal {\n            method: self.method,\n            first: self.first,\n            last: self.last,\n            w_sum: self.weighted_sum,\n        }\n    }\n\n    pub(super) fn interpolate(\n        &self,\n        interval_start: i64,\n        interval_len: i64,\n        prev: Option<TimeWeightSummary>,\n        next: Option<TimeWeightSummary>,\n    ) -> TimeWeightSummary {\n        assert!(\n            interval_start <= self.first.ts,\n            \"Interval start ({}) must be at or before first timestamp ({})\",\n            interval_start,\n            self.first.ts\n        );\n        let end = interval_start + interval_len;\n        assert!(\n            end > self.last.ts,\n            \"Interval end ({}) must be after last timestamp ({})\",\n            end,\n            self.last.ts\n        );\n        let mut new_sum = self.weighted_sum;\n        let new_start = match prev {\n            Some(prev) if interval_start < self.first.ts => {\n                let new_start = self\n                    .method\n                    .interpolate(prev.last, Some(self.first), interval_start)\n                    .expect(\"unable to interpolate start of interval\");\n                new_sum += self.method.weighted_sum(new_start, self.first);\n                new_start\n            }\n            _ => self.first,\n        };\n        let new_end = match (self.method, next) {\n            (_, Some(next)) => {\n                let new_end = self\n                    .method\n                    .interpolate(self.last, Some(next.first), end)\n                    .expect(\"unable to interpolate end of interval\");\n                new_sum += self.method.weighted_sum(self.last, new_end);\n                new_end\n            }\n            (TimeWeightMethod::LOCF, None) => {\n                let new_end = self\n                    .method\n                    .interpolate(self.last, None, end)\n                    .expect(\"unable to interpolate end of interval\");\n                new_sum += self.method.weighted_sum(self.last, new_end);\n                new_end\n            }\n            _ => self.last,\n        };\n\n        unsafe {\n            crate::flatten!(TimeWeightSummary {\n                first: new_start,\n                last: new_end,\n                weighted_sum: new_sum,\n                method: self.method,\n            })\n        }\n    }\n}\n\n#[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]\npub struct TimeWeightTransState {\n    #[serde(skip)]\n    point_buffer: Vec<TSPoint>,\n    method: TimeWeightMethod,\n    summary_buffer: Vec<TimeWeightSummaryInternal>,\n}\n\nimpl TimeWeightTransState {\n    fn push_point(&mut self, value: TSPoint) {\n        self.point_buffer.push(value);\n    }\n\n    fn combine_points(&mut self) {\n        if self.point_buffer.is_empty() {\n            return;\n        }\n        self.point_buffer.sort_unstable_by_key(|p| p.ts);\n        self.summary_buffer.push(\n            TimeWeightSummaryInternal::new_from_sorted_iter(&self.point_buffer, self.method)\n                .unwrap(),\n        );\n        self.point_buffer.clear();\n    }\n\n    fn push_summary(&mut self, other: &TimeWeightTransState) {\n        let cb = other.summary_buffer.clone();\n        for val in cb.into_iter() {\n            self.summary_buffer.push(val);\n        }\n    }\n\n    fn combine_summaries(&mut self) {\n        self.combine_points();\n        if self.summary_buffer.len() <= 1 {\n            return;\n        }\n        self.summary_buffer.sort_unstable_by_key(|s| s.first.ts);\n        self.summary_buffer =\n            vec![TimeWeightSummaryInternal::combine_sorted_iter(&self.summary_buffer).unwrap()];\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn time_weight_trans_serialize(state: Internal) -> bytea {\n    let mut state: Inner<TimeWeightTransState> = unsafe { state.to_inner().unwrap() };\n    state.combine_summaries();\n    crate::do_serialize!(state)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn time_weight_trans_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    time_weight_trans_deserialize_inner(bytes).internal()\n}\npub fn time_weight_trans_deserialize_inner(bytes: bytea) -> Inner<TimeWeightTransState> {\n    let t: TimeWeightTransState = crate::do_deserialize!(bytes, TimeWeightTransState);\n    t.into()\n}\n\n// these are technically parallel_safe (as in they can be called in a parallel context) even though the aggregate itself is parallel restricted.\n#[pg_extern(immutable, parallel_safe)]\npub fn time_weight_trans(\n    state: Internal,\n    method: String,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { time_weight_trans_inner(state.to_inner(), method, ts, val, fcinfo).internal() }\n}\n\npub fn time_weight_trans_inner(\n    state: Option<Inner<TimeWeightTransState>>,\n    method: String,\n    ts: Option<crate::raw::TimestampTz>,\n    val: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<TimeWeightTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let p = match (ts, val) {\n                (_, None) => return state,\n                (None, _) => return state,\n                (Some(ts), Some(val)) => TSPoint { ts: ts.into(), val },\n            };\n\n            match state {\n                None => {\n                    let mut s = TimeWeightTransState {\n                        point_buffer: vec![],\n                        // TODO technically not portable to ASCII-compatible charsets\n                        method: match method.trim().to_lowercase().as_str() {\n                            \"linear\" | \"trapezoidal\" => TimeWeightMethod::Linear,\n                            \"locf\" => TimeWeightMethod::LOCF,\n                            _ => panic!(\"unknown method\"),\n                        },\n                        summary_buffer: vec![],\n                    };\n                    s.push_point(p);\n                    Some(s.into())\n                }\n                Some(mut s) => {\n                    s.push_point(p);\n                    Some(s)\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn time_weight_summary_trans(\n    state: Internal,\n    next: Option<TimeWeightSummary>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    time_weight_summary_trans_inner(unsafe { state.to_inner() }, next, fcinfo).internal()\n}\n\npub fn time_weight_summary_trans_inner(\n    state: Option<Inner<TimeWeightTransState>>,\n    next: Option<TimeWeightSummary>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<TimeWeightTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state, next) {\n            (None, None) => None,\n            (None, Some(next)) => Some(\n                TimeWeightTransState {\n                    summary_buffer: vec![next.internal()],\n                    point_buffer: vec![],\n                    method: next.method,\n                }\n                .into(),\n            ),\n            (Some(state), None) => Some(state),\n            (Some(mut state), Some(next)) => {\n                let next = TimeWeightTransState {\n                    summary_buffer: vec![next.internal()],\n                    point_buffer: vec![],\n                    method: next.method,\n                };\n                state.push_summary(&next);\n                Some(state)\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\npub fn time_weight_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { time_weight_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\n\npub fn time_weight_combine_inner(\n    state1: Option<Inner<TimeWeightTransState>>,\n    state2: Option<Inner<TimeWeightTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<TimeWeightTransState>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            match (state1, state2) {\n                (None, None) => None,\n                (None, Some(state2)) => {\n                    let mut s = state2.clone();\n                    s.combine_points();\n                    Some(s.into())\n                }\n                (Some(state1), None) => {\n                    let mut s = state1.clone();\n                    s.combine_points();\n                    Some(s.into())\n                }\n                (Some(state1), Some(state2)) => {\n                    let mut s1 = state1.clone(); // is there a way to avoid if it doesn't need it?\n                    s1.combine_points();\n                    let mut s2 = state2.clone();\n                    s2.combine_points();\n                    s2.push_summary(&s1);\n                    Some(s2.into())\n                }\n            }\n        })\n    }\n}\n\n#[pg_extern(immutable, parallel_safe)]\nfn time_weight_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<TimeWeightSummary> {\n    time_weight_final_inner(unsafe { state.to_inner() }, fcinfo)\n}\n\nfn time_weight_final_inner(\n    state: Option<Inner<TimeWeightTransState>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<TimeWeightSummary> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let mut state = match state {\n                None => return None,\n                Some(state) => state.clone(),\n            };\n            state.combine_summaries();\n            debug_assert!(state.summary_buffer.len() <= 1);\n            state.summary_buffer.pop().map(|st| {\n                flatten!(TimeWeightSummary {\n                    method: st.method,\n                    first: st.first,\n                    last: st.last,\n                    weighted_sum: st.w_sum,\n                })\n            })\n        })\n    }\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weight_first_val(sketch: TimeWeightSummary, _accessor: AccessorFirstVal) -> f64 {\n    time_weight_first_val(sketch)\n}\n\n#[pg_extern(name = \"first_val\", strict, immutable, parallel_safe)]\nfn time_weight_first_val(summary: TimeWeightSummary) -> f64 {\n    summary.first.val\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weight_last_val(sketch: TimeWeightSummary, _accessor: AccessorLastVal) -> f64 {\n    time_weight_last_val(sketch)\n}\n\n#[pg_extern(name = \"last_val\", strict, immutable, parallel_safe)]\nfn time_weight_last_val(summary: TimeWeightSummary) -> f64 {\n    summary.last.val\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weight_first_time(\n    sketch: TimeWeightSummary,\n    _accessor: AccessorFirstTime,\n) -> crate::raw::TimestampTz {\n    time_weight_first_time(sketch)\n}\n\n#[pg_extern(name = \"first_time\", strict, immutable, parallel_safe)]\nfn time_weight_first_time(summary: TimeWeightSummary) -> crate::raw::TimestampTz {\n    summary.first.ts.into()\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weight_last_time(\n    sketch: TimeWeightSummary,\n    _accessor: AccessorLastTime,\n) -> crate::raw::TimestampTz {\n    time_weight_last_time(sketch)\n}\n\n#[pg_extern(name = \"last_time\", strict, immutable, parallel_safe)]\nfn time_weight_last_time(summary: TimeWeightSummary) -> crate::raw::TimestampTz {\n    summary.last.ts.into()\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE time_weight(method text, ts timestamptz, value DOUBLE PRECISION)\\n\\\n    (\\n\\\n        sfunc = time_weight_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = time_weight_final,\\n\\\n        combinefunc = time_weight_combine,\\n\\\n        serialfunc = time_weight_trans_serialize,\\n\\\n        deserialfunc = time_weight_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\\\n\\n\\\n    CREATE AGGREGATE rollup(tws TimeWeightSummary)\\n\\\n    (\\n\\\n        sfunc = time_weight_summary_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = time_weight_final,\\n\\\n        combinefunc = time_weight_combine,\\n\\\n        serialfunc = time_weight_trans_serialize,\\n\\\n        deserialfunc = time_weight_trans_deserialize,\\n\\\n        parallel = restricted\\n\\\n    );\\n\\\n\",\n    name = \"time_weight_agg\",\n    requires = [\n        time_weight_trans,\n        time_weight_final,\n        time_weight_combine,\n        time_weight_trans_serialize,\n        time_weight_trans_deserialize,\n        time_weight_summary_trans\n    ],\n);\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weighted_average_average(\n    sketch: Option<TimeWeightSummary>,\n    _accessor: AccessorAverage,\n) -> Option<f64> {\n    time_weighted_average_average(sketch)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weighted_average_integral(\n    tws: Option<TimeWeightSummary>,\n    accessor: AccessorIntegral,\n) -> Option<f64> {\n    time_weighted_average_integral(\n        tws,\n        String::from_utf8_lossy(&accessor.bytes[..accessor.len as usize]).to_string(),\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"average\")]\npub fn time_weighted_average_average(tws: Option<TimeWeightSummary>) -> Option<f64> {\n    match tws {\n        None => None,\n        Some(tws) => match tws.internal().time_weighted_average() {\n            Ok(a) => Some(a),\n            //without bounds, the average for a single value is undefined, but it probably shouldn't throw an error, we'll return null for now.\n            Err(e) => {\n                if e == TimeWeightError::ZeroDuration {\n                    None\n                } else {\n                    Err(e).unwrap()\n                }\n            }\n        },\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"integral\")]\npub fn time_weighted_average_integral(\n    tws: Option<TimeWeightSummary>,\n    unit: default!(String, \"'second'\"),\n) -> Option<f64> {\n    let unit = match DurationUnit::from_str(&unit) {\n        Some(unit) => unit,\n        None => pgrx::error!(\n            \"Unrecognized duration unit: {}. Valid units are: usecond, msecond, second, minute, hour\",\n            unit,\n        ),\n    };\n    let integral_microsecs = tws?.internal().time_weighted_integral();\n    Some(DurationUnit::Microsec.convert_unit(integral_microsecs, unit))\n}\n\nfn interpolate(\n    tws: Option<TimeWeightSummary>,\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: Option<TimeWeightSummary>,\n    next: Option<TimeWeightSummary>,\n) -> Option<TimeWeightSummary> {\n    match tws {\n        None => None,\n        Some(tws) => {\n            let interval = crate::datum_utils::interval_to_ms(&start, &duration);\n            Some(tws.interpolate(start.into(), interval, prev, next))\n        }\n    }\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_average\")]\npub fn time_weighted_average_interpolated_average(\n    tws: Option<TimeWeightSummary>,\n    start: crate::raw::TimestampTz,\n    duration: crate::raw::Interval,\n    prev: default!(Option<TimeWeightSummary>, \"NULL\"),\n    next: default!(Option<TimeWeightSummary>, \"NULL\"),\n) -> Option<f64> {\n    let target = interpolate(tws, start, duration, prev, next);\n    time_weighted_average_average(target)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weighted_average_interpolated_average(\n    tws: Option<TimeWeightSummary>,\n    accessor: TimeWeightInterpolatedAverageAccessor,\n) -> Option<f64> {\n    let prev = if accessor.flags & 1 == 1 {\n        Some(accessor.prev.clone().into())\n    } else {\n        None\n    };\n    let next = if accessor.flags & 2 == 2 {\n        Some(accessor.next.clone().into())\n    } else {\n        None\n    };\n\n    time_weighted_average_interpolated_average(\n        tws,\n        accessor.timestamp.into(),\n        accessor.interval.into(),\n        prev,\n        next,\n    )\n}\n\n#[pg_extern(immutable, parallel_safe, name = \"interpolated_integral\")]\npub fn time_weighted_average_interpolated_integral(\n    tws: Option<TimeWeightSummary>,\n    start: crate::raw::TimestampTz,\n    interval: crate::raw::Interval,\n    prev: default!(Option<TimeWeightSummary>, \"NULL\"),\n    next: default!(Option<TimeWeightSummary>, \"NULL\"),\n    unit: default!(String, \"'second'\"),\n) -> Option<f64> {\n    let target = interpolate(tws, start, interval, prev, next);\n    time_weighted_average_integral(target, unit)\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_time_weighted_average_interpolated_integral(\n    tws: Option<TimeWeightSummary>,\n    accessor: TimeWeightInterpolatedIntegralAccessor,\n) -> Option<f64> {\n    let prev = if accessor.flags & 1 == 1 {\n        Some(accessor.prev.clone().into())\n    } else {\n        None\n    };\n    let next = if accessor.flags & 2 == 2 {\n        Some(accessor.next.clone().into())\n    } else {\n        None\n    };\n\n    // Convert from num of milliseconds to DurationUnit and then to string\n    let unit = match accessor.unit {\n        1 => DurationUnit::Microsec,\n        1000 => DurationUnit::Millisec,\n        1_000_000 => DurationUnit::Second,\n        60_000_000 => DurationUnit::Minute,\n        3_600_000_000 => DurationUnit::Hour,\n        _ => todo!(), // This should never be reached, the accessor gets these numbers from microseconds() in duration.rs, which only matches on valid enum values\n    }\n    .to_string();\n\n    time_weighted_average_interpolated_integral(\n        tws,\n        accessor.start.into(),\n        accessor.interval.into(),\n        prev,\n        next,\n        unit,\n    )\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n\n    use pgrx_macros::pg_test;\n    macro_rules! select_one {\n        ($client:expr, $stmt:expr, $type:ty) => {\n            $client\n                .update($stmt, None, &[])\n                .unwrap()\n                .first()\n                .get_one::<$type>()\n                .unwrap()\n                .unwrap()\n        };\n    }\n    #[pg_test]\n    fn test_time_weight_aggregate() {\n        Spi::connect_mut(|client| {\n            let stmt =\n                \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION); SET TIME ZONE 'UTC'\";\n            client.update(stmt, None, &[]).unwrap();\n\n            // add a point\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:00:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"SELECT integral(time_weight('Trapezoidal', ts, val), 'hrs') FROM test\";\n            assert_eq!(select_one!(client, stmt, f64), 0.0);\n            let stmt = \"SELECT integral(time_weight('LOCF', ts, val), 'msecond') FROM test\";\n            assert_eq!(select_one!(client, stmt, f64), 0.0);\n\n            // add another point\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:01:00+00', 20.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            // test basic with 2 points\n            let stmt = \"SELECT average(time_weight('Linear', ts, val)) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 15.0).abs() < f64::EPSILON);\n            let stmt = \"SELECT average(time_weight('LOCF', ts, val)) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 10.0).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT first_val(time_weight('LOCF', ts, val)) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 10.0).abs() < f64::EPSILON);\n            let stmt = \"SELECT last_val(time_weight('LOCF', ts, val)) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 20.0).abs() < f64::EPSILON);\n\n            // arrow syntax should be the same\n            let stmt = \"SELECT time_weight('LOCF', ts, val) -> first_val() FROM test\";\n            assert!((select_one!(client, stmt, f64) - 10.0).abs() < f64::EPSILON);\n            let stmt = \"SELECT time_weight('LOCF', ts, val) -> last_val() FROM test\";\n            assert!((select_one!(client, stmt, f64) - 20.0).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT first_time(time_weight('LOCF', ts, val))::text FROM test\";\n            assert_eq!(select_one!(client, stmt, &str), \"2020-01-01 00:00:00+00\");\n            let stmt = \"SELECT last_time(time_weight('LOCF', ts, val))::text FROM test\";\n            assert_eq!(select_one!(client, stmt, &str), \"2020-01-01 00:01:00+00\");\n\n            // arrow syntax should be the same\n            let stmt = \"SELECT (time_weight('LOCF', ts, val) -> first_time())::text FROM test\";\n            assert_eq!(select_one!(client, stmt, &str), \"2020-01-01 00:00:00+00\");\n            let stmt = \"SELECT (time_weight('LOCF', ts, val) -> last_time())::text FROM test\";\n            assert_eq!(select_one!(client, stmt, &str), \"2020-01-01 00:01:00+00\");\n\n            // more values evenly spaced\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:02:00+00', 10.0), ('2020-01-01 00:03:00+00', 20.0), ('2020-01-01 00:04:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"SELECT average(time_weight('Linear', ts, val)) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 15.0).abs() < f64::EPSILON);\n            let stmt = \"SELECT average(time_weight('LOCF', ts, val)) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 15.0).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT integral(time_weight('Linear', ts, val), 'mins') FROM test\";\n            assert!((select_one!(client, stmt, f64) - 60.0).abs() < f64::EPSILON);\n            let stmt = \"SELECT integral(time_weight('LOCF', ts, val), 'hour') FROM test\";\n            assert!((select_one!(client, stmt, f64) - 1.0).abs() < f64::EPSILON);\n\n            //non-evenly spaced values\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:08:00+00', 30.0), ('2020-01-01 00:10:00+00', 10.0), ('2020-01-01 00:10:30+00', 20.0), ('2020-01-01 00:20:00+00', 30.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let stmt = \"SELECT average(time_weight('Linear', ts, val)) FROM test\";\n            // expected =(15 +15 +15 +15 + 20*4 + 20*2 +15*.5 + 25*9.5) / 20 = 21.25 just taking the midpoints between each point and multiplying by minutes and dividing by total\n            assert!((select_one!(client, stmt, f64) - 21.25).abs() < f64::EPSILON);\n            let stmt = \"SELECT time_weight('Linear', ts, val) \\\n                ->average() \\\n            FROM test\";\n            // arrow syntax should be the same\n            assert!((select_one!(client, stmt, f64) - 21.25).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT integral(time_weight('Linear', ts, val), 'microseconds') FROM test\";\n            assert!((select_one!(client, stmt, f64) - 25500000000.00).abs() < f64::EPSILON);\n            let stmt = \"SELECT time_weight('Linear', ts, val) \\\n                ->integral('microseconds') \\\n            FROM test\";\n            // arrow syntax should be the same\n            assert!((select_one!(client, stmt, f64) - 25500000000.00).abs() < f64::EPSILON);\n            let stmt = \"SELECT time_weight('Linear', ts, val) \\\n                ->integral() \\\n            FROM test\";\n            assert!((select_one!(client, stmt, f64) - 25500.00).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT average(time_weight('LOCF', ts, val)) FROM test\";\n            // expected = (10 + 20 + 10 + 20 + 10*4 + 30*2 +10*.5 + 20*9.5) / 20 = 17.75 using last value and carrying for each point\n            assert!((select_one!(client, stmt, f64) - 17.75).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT integral(time_weight('LOCF', ts, val), 'milliseconds') FROM test\";\n            assert!((select_one!(client, stmt, f64) - 21300000.0).abs() < f64::EPSILON);\n\n            //make sure this works with whatever ordering we throw at it\n            let stmt = \"SELECT average(time_weight('Linear', ts, val ORDER BY random())) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 21.25).abs() < f64::EPSILON);\n            let stmt = \"SELECT average(time_weight('LOCF', ts, val ORDER BY random())) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 17.75).abs() < f64::EPSILON);\n\n            let stmt = \"SELECT integral(time_weight('Linear', ts, val ORDER BY random()), 'seconds') FROM test\";\n            assert!((select_one!(client, stmt, f64) - 25500.0).abs() < f64::EPSILON);\n            let stmt = \"SELECT integral(time_weight('LOCF', ts, val ORDER BY random())) FROM test\";\n            assert!((select_one!(client, stmt, f64) - 21300.0).abs() < f64::EPSILON);\n\n            // make sure we get the same result if we do multi-level aggregation\n            let stmt = \"WITH t AS (SELECT date_trunc('minute', ts), time_weight('Linear', ts, val) AS tws FROM test GROUP BY 1) SELECT average(rollup(tws)) FROM t\";\n            assert!((select_one!(client, stmt, f64) - 21.25).abs() < f64::EPSILON);\n            let stmt = \"WITH t AS (SELECT date_trunc('minute', ts), time_weight('LOCF', ts, val) AS tws FROM test GROUP BY 1) SELECT average(rollup(tws)) FROM t\";\n            assert!((select_one!(client, stmt, f64) - 17.75).abs() < f64::EPSILON);\n        });\n    }\n\n    #[pg_test]\n    fn test_time_weight_io() {\n        Spi::connect_mut(|client| {\n            client.update(\"SET timezone TO 'UTC'\", None, &[]).unwrap();\n            let stmt = \"CREATE TABLE test(ts timestamptz, val DOUBLE PRECISION)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let linear_time_weight = \"SELECT time_weight('Linear', ts, val)::TEXT FROM test\";\n            let locf_time_weight = \"SELECT time_weight('LOCF', ts, val)::TEXT FROM test\";\n            let avg = |text: &str| format!(\"SELECT average('{text}'::TimeWeightSummary)\");\n\n            // add a couple points\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:00:00+00', 10.0), ('2020-01-01 00:01:00+00', 20.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            // test basic with 2 points\n            let expected = \"(\\\n                version:1,\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                last:(ts:\\\"2020-01-01 00:01:00+00\\\",val:20),\\\n                weighted_sum:900000000,\\\n                method:Linear\\\n            )\";\n            assert_eq!(select_one!(client, linear_time_weight, String), expected);\n            assert!((select_one!(client, &*avg(expected), f64) - 15.0).abs() < f64::EPSILON);\n\n            let expected = \"(\\\n                version:1,\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                last:(ts:\\\"2020-01-01 00:01:00+00\\\",val:20),\\\n                weighted_sum:600000000,\\\n                method:LOCF\\\n            )\";\n            assert_eq!(select_one!(client, locf_time_weight, String), expected);\n            assert!((select_one!(client, &*avg(expected), f64) - 10.0).abs() < f64::EPSILON);\n\n            // more values evenly spaced\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:02:00+00', 10.0), ('2020-01-01 00:03:00+00', 20.0), ('2020-01-01 00:04:00+00', 10.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                last:(ts:\\\"2020-01-01 00:04:00+00\\\",val:10),\\\n                weighted_sum:3600000000,\\\n                method:Linear\\\n            )\";\n            assert_eq!(select_one!(client, linear_time_weight, String), expected);\n            assert!((select_one!(client, &*avg(expected), f64) - 15.0).abs() < f64::EPSILON);\n            let expected = \"(\\\n                version:1,\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                last:(ts:\\\"2020-01-01 00:04:00+00\\\",val:10),\\\n                weighted_sum:3600000000,\\\n                method:LOCF\\\n            )\";\n            assert_eq!(select_one!(client, locf_time_weight, String), expected);\n            assert!((select_one!(client, &*avg(expected), f64) - 15.0).abs() < f64::EPSILON);\n\n            //non-evenly spaced values\n            let stmt = \"INSERT INTO test VALUES('2020-01-01 00:08:00+00', 30.0), ('2020-01-01 00:10:00+00', 10.0), ('2020-01-01 00:10:30+00', 20.0), ('2020-01-01 00:20:00+00', 30.0)\";\n            client.update(stmt, None, &[]).unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                last:(ts:\\\"2020-01-01 00:20:00+00\\\",val:30),\\\n                weighted_sum:25500000000,\\\n                method:Linear\\\n            )\";\n            assert_eq!(select_one!(client, linear_time_weight, String), expected);\n            assert!((select_one!(client, &*avg(expected), f64) - 21.25).abs() < f64::EPSILON);\n            let expected = \"(\\\n                version:1,\\\n                first:(ts:\\\"2020-01-01 00:00:00+00\\\",val:10),\\\n                last:(ts:\\\"2020-01-01 00:20:00+00\\\",val:30),\\\n                weighted_sum:21300000000,\\\n                method:LOCF\\\n            )\";\n            assert_eq!(select_one!(client, locf_time_weight, String), expected);\n            assert!((select_one!(client, &*avg(expected), f64) - 17.75).abs() < f64::EPSILON);\n        });\n    }\n\n    #[pg_test]\n    fn test_time_weight_byte_io() {\n        unsafe {\n            use std::ptr;\n            const BASE: i64 = 631152000000000;\n            const MIN: i64 = 60000000;\n            let state = time_weight_trans_inner(\n                None,\n                \"linear\".to_string(),\n                Some(BASE.into()),\n                Some(10.0),\n                ptr::null_mut(),\n            );\n            let state = time_weight_trans_inner(\n                state,\n                \"linear\".to_string(),\n                Some((BASE + MIN).into()),\n                Some(20.0),\n                ptr::null_mut(),\n            );\n            let state = time_weight_trans_inner(\n                state,\n                \"linear\".to_string(),\n                Some((BASE + 2 * MIN).into()),\n                Some(30.0),\n                ptr::null_mut(),\n            );\n            let state = time_weight_trans_inner(\n                state,\n                \"linear\".to_string(),\n                Some((BASE + 3 * MIN).into()),\n                Some(10.0),\n                ptr::null_mut(),\n            );\n            let state = time_weight_trans_inner(\n                state,\n                \"linear\".to_string(),\n                Some((BASE + 4 * MIN).into()),\n                Some(20.0),\n                ptr::null_mut(),\n            );\n            let state = time_weight_trans_inner(\n                state,\n                \"linear\".to_string(),\n                Some((BASE + 5 * MIN).into()),\n                Some(30.0),\n                ptr::null_mut(),\n            );\n\n            let mut control = state.unwrap();\n            let buffer =\n                time_weight_trans_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = pgrx::varlena::varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let expected = [\n                1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 96, 194, 134, 7, 62, 2, 0,\n                0, 0, 0, 0, 0, 0, 36, 64, 0, 3, 164, 152, 7, 62, 2, 0, 0, 0, 0, 0, 0, 0, 62, 64, 0,\n                0, 0, 192, 11, 90, 246, 65,\n            ];\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let new_state =\n                time_weight_trans_deserialize_inner(bytea(pg_sys::Datum::from(expected.as_ptr())));\n\n            control.combine_summaries(); // Serialized form is always combined\n            assert_eq!(&*new_state, &*control);\n        }\n    }\n\n    #[pg_test]\n    fn test_time_weight_interpolation() {\n        Spi::connect_mut(|client| {\n            client.update(\n                \"CREATE TABLE test(time timestamptz, value double precision, bucket timestamptz)\",\n                None,\n                &[]\n            ).unwrap();\n            client\n                .update(\n                    r#\"INSERT INTO test VALUES\n                ('2020-1-1 8:00'::timestamptz, 10.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 12:00'::timestamptz, 40.0, '2020-1-1'::timestamptz),\n                ('2020-1-1 16:00'::timestamptz, 20.0, '2020-1-1'::timestamptz),\n                ('2020-1-2 2:00'::timestamptz, 15.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 12:00'::timestamptz, 50.0, '2020-1-2'::timestamptz),\n                ('2020-1-2 20:00'::timestamptz, 25.0, '2020-1-2'::timestamptz),\n                ('2020-1-3 10:00'::timestamptz, 30.0, '2020-1-3'::timestamptz),\n                ('2020-1-3 12:00'::timestamptz, 0.0, '2020-1-3'::timestamptz), \n                ('2020-1-3 16:00'::timestamptz, 35.0, '2020-1-3'::timestamptz)\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let mut averages = client\n                .update(\n                    r#\"SELECT\n                interpolated_average(\n                    agg,\n                    bucket,\n                    '1 day'::interval,\n                    LAG(agg) OVER (ORDER BY bucket),\n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, time_weight('LOCF', time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            // test arrow version\n            let mut arrow_averages = client\n                .update(\n                    r#\"SELECT\n                agg -> interpolated_average(\n                    bucket,\n                    '1 day'::interval,\n                    LAG(agg) OVER (ORDER BY bucket),\n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, time_weight('LOCF', time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            let mut integrals = client\n                .update(\n                    r#\"SELECT\n                interpolated_integral(\n                    agg,\n                    bucket,\n                    '1 day'::interval, \n                    LAG(agg) OVER (ORDER BY bucket),\n                    LEAD(agg) OVER (ORDER BY bucket),\n                    'hours'\n                ) FROM (\n                    SELECT bucket, time_weight('LOCF', time, value) as agg \n                    FROM test \n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n            // verify that default value works\n            client\n                .update(\n                    r#\"SELECT\n                interpolated_integral(\n                    agg,\n                    bucket,\n                    '1 day'::interval,\n                    LAG(agg) OVER (ORDER BY bucket),\n                    LEAD(agg) OVER (ORDER BY bucket)\n                ) FROM (\n                    SELECT bucket, time_weight('LOCF', time, value) as agg\n                    FROM test\n                    GROUP BY bucket\n                ) s\n                ORDER BY bucket\"#,\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            // Day 1, 4 hours @ 10, 4 @ 40, 8 @ 20\n            let result = averages.next().unwrap()[1].value().unwrap();\n            assert_eq!(result, Some((4. * 10. + 4. * 40. + 8. * 20.) / 16.));\n            assert_eq!(result, arrow_averages.next().unwrap()[1].value().unwrap());\n\n            assert_eq!(\n                integrals.next().unwrap()[1].value().unwrap(),\n                Some(4. * 10. + 4. * 40. + 8. * 20.)\n            );\n            // Day 2, 2 hours @ 20, 10 @ 15, 8 @ 50, 4 @ 25\n            let result = averages.next().unwrap()[1].value().unwrap();\n            assert_eq!(\n                result,\n                Some((2. * 20. + 10. * 15. + 8. * 50. + 4. * 25.) / 24.)\n            );\n            assert_eq!(result, arrow_averages.next().unwrap()[1].value().unwrap());\n            assert_eq!(\n                integrals.next().unwrap()[1].value().unwrap(),\n                Some(2. * 20. + 10. * 15. + 8. * 50. + 4. * 25.)\n            );\n            // Day 3, 10 hours @ 25, 2 @ 30, 4 @ 0, 8 @ 35\n            let result = averages.next().unwrap()[1].value().unwrap();\n            assert_eq!(result, Some((10. * 25. + 2. * 30. + 8. * 35.) / 24.));\n            assert_eq!(result, arrow_averages.next().unwrap()[1].value().unwrap());\n            assert_eq!(\n                integrals.next().unwrap()[1].value().unwrap(),\n                Some(10. * 25. + 2. * 30. + 8. * 35.)\n            );\n            assert!(averages.next().is_none());\n            assert!(arrow_averages.next().is_none());\n            assert!(integrals.next().is_none());\n        });\n    }\n\n    #[pg_test]\n    fn test_locf_interpolation_to_null() {\n        Spi::connect_mut(|client| {\n            let stmt =\n                \"SELECT interpolated_average(time_weight('locf', '2020-01-01 20:00:00+00', 100),\n                    '2020-01-01 00:00:00+00', '1d')\";\n            assert_eq!(select_one!(client, stmt, f64), 100.0);\n            let stmt = \"SELECT time_weight('locf', '2020-01-01 20:00:00+00', 100)\n                 -> interpolated_integral('2020-01-01 00:00:00+00', '1d')\";\n            assert_eq!(select_one!(client, stmt, f64), 1440000.0);\n        });\n    }\n}\n"
  },
  {
    "path": "extension/src/type_builder.rs",
    "content": "#[derive(Copy, Clone, Debug, serde::Serialize)]\npub enum CachedDatum<'r> {\n    None,\n    FromInput(&'r [u8]),\n    Flattened(&'r [u8]),\n}\n\nimpl PartialEq for CachedDatum<'_> {\n    fn eq(&self, _: &Self) -> bool {\n        true\n    }\n}\n\n// XXX Required by [`pgrx::PostgresType`] for default [`pgrx::FromDatum`]\n// implementation but isn't used since we implement [`pgrx::FromDatum`]\n// ourselves. We need a custom implementation because with the default one the\n// compiler complains that `'input` and `'de` lifetimes are incompatible.\nimpl<'de> serde::Deserialize<'de> for CachedDatum<'_> {\n    fn deserialize<D>(_deserializer: D) -> Result<Self, D::Error>\n    where\n        D: serde::de::Deserializer<'de>,\n    {\n        unimplemented!();\n    }\n}\n\n// Routes to pg_type_impl! for lifetime types, pg_type_no_lifetime_impl! for non-lifetime types\n#[macro_export]\nmacro_rules! pg_type {\n    // BASE CASE: Struct WITH lifetimes → route to pg_type_impl!\n    (\n        $(#[$attrs: meta])*\n        struct $name: ident<$($inlife: lifetime),+> {\n            $(,)?\n        }\n\n        $(%($($vals:tt)*))?\n    ) => {\n        $crate::pg_type_impl!{\n            'input\n            $(#[$attrs])*\n            struct $name<$($inlife),+>\n            {\n                $($($vals)*)?\n            }\n        }\n    };\n\n    // BASE CASE: Struct WITHOUT lifetimes → route to pg_type_no_lifetime_impl!\n    (\n        $(#[$attrs: meta])*\n        struct $name: ident {\n            $(,)?\n        }\n\n        $(%($($vals:tt)*))?\n    ) => {\n        $crate::pg_type_no_lifetime_impl!{\n            $(#[$attrs])*\n            struct $name\n            {\n                $($($vals)*)?\n            }\n        }\n    };\n\n    // FIELD PROCESSING: Struct WITH lifetimes\n    // Recurses into this very same macro\n    (\n        $(#[$attrs: meta])*\n        struct $name: ident<$($inlife: lifetime),+> {\n            $(#[$fattrs: meta])* $field:ident : $typ: tt $(<$life:lifetime>)?,\n            $($tail: tt)*\n        }\n\n        $(%($($vals:tt)*))?\n    ) => {\n        $crate::pg_type!{\n            $(#[$attrs])*\n            struct $name<$($inlife),+> {\n                $($tail)*\n            }\n\n            %( $($($vals)*)?\n                $(#[$fattrs])* $field : $typ $(<$life>)? ,\n            )\n        }\n    };\n\n    // FIELD PROCESSING: Struct WITHOUT lifetimes\n    // Recurses into this very same macro\n    (\n        $(#[$attrs: meta])*\n        struct $name: ident {\n            $(#[$fattrs: meta])* $field:ident : $typ: ty,\n            $($tail: tt)*\n        }\n\n        $(%($($vals:tt)*))?\n    ) => {\n        $crate::pg_type!{\n            $(#[$attrs])*\n            struct $name {\n                $($tail)*\n            }\n\n            %( $($($vals)*)?\n                $(#[$fattrs])* $field : $typ ,\n            )\n        }\n    };\n}\n\n#[macro_export]\nmacro_rules! pg_type_impl {\n    (\n        $lifetemplate: lifetime\n        $(#[$attrs: meta])*\n        struct $name: ident $(<$inlife: lifetime>)? {\n            $($(#[$fattrs: meta])* $field:ident : $typ: tt $(<$life:lifetime>)?),*\n            $(,)?\n        }\n    ) => {\n        ::paste::paste! {\n            // This is where the main difference between the lifetime and no-lifetime versions is\n            // that the generated struct DOES NOT derive form PostgresType. This is because the\n            // lifetime support in the derive macro is fundamentally broken because some generated\n            // functions don't have an input parameter where the lifetime parameter can be bound to.\n            $(#[$attrs])*\n            #[derive(Clone, serde::Serialize, serde::Deserialize)]\n            pub struct $name<$lifetemplate>(pub [<$name Data>] $(<$inlife>)?, $crate::type_builder::CachedDatum<$lifetemplate>);\n\n            // Manual PostgresType implementation - the derive macro can't handle lifetimes in pgrx 0.16.0\n            impl<$lifetemplate> ::pgrx::datum::PostgresType for $name<$lifetemplate> {}\n\n            unsafe impl<$lifetemplate> ::pgrx_sql_entity_graph::metadata::SqlTranslatable for $name<$lifetemplate> {\n                fn argument_sql() -> core::result::Result<::pgrx_sql_entity_graph::metadata::SqlMapping, ::pgrx_sql_entity_graph::metadata::ArgumentError> {\n                    Ok(::pgrx_sql_entity_graph::metadata::SqlMapping::As(String::from(stringify!($name))))\n                }\n\n                fn return_sql() -> core::result::Result<::pgrx_sql_entity_graph::metadata::Returns, ::pgrx_sql_entity_graph::metadata::ReturnsError> {\n                    Ok(::pgrx_sql_entity_graph::metadata::Returns::One(::pgrx_sql_entity_graph::metadata::SqlMapping::As(String::from(stringify!($name)))))\n                }\n            }\n\n            ::paste::paste! {\n                #[unsafe(no_mangle)]\n                #[doc(hidden)]\n                #[allow(nonstandard_style, unknown_lints, clippy::no_mangle_with_rust_abi)]\n                pub extern \"Rust\" fn [<__pgrx_internals_type_ $name>]() -> ::pgrx_sql_entity_graph::SqlGraphEntity {\n                    extern crate alloc;\n                    use alloc::string::ToString;\n                    use ::pgrx::datum::WithTypeIds;\n\n                    let mut mappings = Default::default();\n                    <$name<'_> as ::pgrx::datum::WithTypeIds>::register_with_refs(\n                        &mut mappings,\n                        stringify!($name).to_string()\n                    );\n                    ::pgrx::datum::WithSizedTypeIds::<$name<'_>>::register_sized_with_refs(\n                        &mut mappings,\n                        stringify!($name).to_string()\n                    );\n                    ::pgrx::datum::WithArrayTypeIds::<$name<'_>>::register_array_with_refs(\n                        &mut mappings,\n                        stringify!($name).to_string()\n                    );\n                    ::pgrx::datum::WithVarlenaTypeIds::<$name<'_>>::register_varlena_with_refs(\n                        &mut mappings,\n                        stringify!($name).to_string()\n                    );\n\n                    let submission = ::pgrx_sql_entity_graph::PostgresTypeEntity {\n                        name: stringify!($name),\n                        file: file!(),\n                        line: line!(),\n                        module_path: module_path!(),\n                        full_path: core::any::type_name::<$name<'_>>(),\n                        mappings: mappings.into_iter().collect(),\n                        in_fn: stringify!([<$name:lower _in>]),\n                        in_fn_module_path: module_path!().to_string(),\n                        out_fn: stringify!([<$name:lower _out>]),\n                        out_fn_module_path: module_path!().to_string(),\n                        receive_fn: None,\n                        receive_fn_module_path: None,\n                        send_fn: None,\n                        send_fn_module_path: None,\n                        to_sql_config: ::pgrx::pgrx_sql_entity_graph::ToSqlConfigEntity {\n                            enabled: true,\n                            callback: None,\n                            content: None,\n                        },\n                        alignment: None,\n                    };\n                    ::pgrx_sql_entity_graph::SqlGraphEntity::Type(submission)\n                }\n            }\n\n            #[doc(hidden)]\n            #[::pgrx::pgrx_macros::pg_extern(immutable, parallel_safe)]\n            pub fn [<$name:lower _in>](input: Option<&::core::ffi::CStr>) -> Option<$name<'static>> {\n                input.map_or_else(|| {\n                    if let Some(m) = <$name<'static> as ::pgrx::inoutfuncs::InOutFuncs>::NULL_ERROR_MESSAGE {\n                        ::pgrx::pg_sys::error!(\"{m}\");\n                    }\n                    None\n                }, |i| Some(<$name<'static> as ::pgrx::inoutfuncs::InOutFuncs>::input(i)))\n            }\n\n            #[doc(hidden)]\n            #[::pgrx::pgrx_macros::pg_extern(immutable, parallel_safe)]\n            pub fn [<$name:lower _out>](input: $name<'static>) -> ::pgrx::ffi::CString {\n                let mut buffer = ::pgrx::stringinfo::StringInfo::new();\n                ::pgrx::inoutfuncs::InOutFuncs::output(&input, &mut buffer);\n                // SAFETY: We just constructed this StringInfo ourselves\n                unsafe { buffer.leak_cstr().to_owned() }\n            }\n\n            flat_serialize_macro::flat_serialize! {\n                $(#[$attrs])*\n                #[derive(serde::Serialize, serde::Deserialize)]\n                struct [<$name Data>] $(<$inlife>)? {\n                    #[serde(skip, default=\"crate::serialization::serde_reference_adaptor::default_header\")]\n                    header: u32,\n                    version: u8,\n                    #[serde(skip, default=\"crate::serialization::serde_reference_adaptor::default_padding\")]\n                    padding: [u8; 3],\n                    $($(#[$fattrs])* $field: $typ $(<$life>)?),*\n                }\n            }\n\n            impl<'input> $name<'input> {\n                pub fn in_current_context<'foo>(&self) -> $name<'foo> {\n                    unsafe { self.0.flatten() }\n                }\n\n                #[allow(clippy::missing_safety_doc)]\n                pub unsafe fn cached_datum_or_flatten(&mut self) -> pgrx::pg_sys::Datum {\n                    use $crate::type_builder::CachedDatum::*;\n                    match self.1 {\n                        None => {\n                            *self = unsafe { self.0.flatten() };\n                            unsafe { self.cached_datum_or_flatten() }\n                        },\n                        FromInput(bytes) | Flattened(bytes) => pg_sys::Datum::from(bytes.as_ptr()),\n                    }\n                }\n            }\n\n            impl<$lifetemplate> [<$name Data>] $(<$inlife>)? {\n                #[allow(clippy::missing_safety_doc)]\n                pub unsafe fn flatten<'any>(&self) -> $name<'any> {\n                    use flat_serialize::FlatSerializable as _;\n                    use $crate::type_builder::CachedDatum::Flattened;\n                    // if we already have a CachedDatum::Flattened can just\n                    // return it without re-flattening?\n                    // TODO this needs extensive testing before we enable it\n                    //  XXX this will not work if the lifetime of the memory\n                    //      context the value was previously flattened into is\n                    //      wrong; this may be bad enough that we should never\n                    //      enable it by default...\n                    // if let Flattened(bytes) = self.1 {\n                    //     let bytes = extend_lifetime(bytes);\n                    //     let wrapped = [<$name Data>]::try_ref(bytes).unwrap().0;\n                    //     $name(wrapped, Flattened(bytes))\n                    //     return self\n                    // }\n                    let bytes: &'static [u8] = self.to_pg_bytes();\n                    let wrapped = [<$name Data>]::try_ref(bytes).unwrap().0;\n                    $name(wrapped, Flattened(bytes))\n                }\n\n                pub fn to_pg_bytes(&self) -> &'static [u8] {\n                    use std::{mem::MaybeUninit, slice};\n                    use flat_serialize::FlatSerializable as _;\n                    unsafe {\n                        let len = self.num_bytes();\n                        // valena types have a maximum size\n                        if len > 0x3FFFFFFF {\n                            pgrx::error!(\"size {} bytes is to large\", len)\n                        }\n                        let memory: *mut MaybeUninit<u8> = pg_sys::palloc0(len).cast();\n                        let slice = slice::from_raw_parts_mut(memory, len);\n                        let rem = self.fill_slice(slice);\n                        debug_assert_eq!(rem.len(), 0);\n\n                        ::pgrx::set_varsize_4b(memory.cast(), len as i32);\n                        slice::from_raw_parts(memory.cast(), len)\n                    }\n                }\n            }\n\n            impl<$lifetemplate> pgrx::FromDatum for $name<$lifetemplate> {\n                unsafe fn from_polymorphic_datum(datum: pgrx::pg_sys::Datum, is_null: bool, _: pg_sys::Oid) -> Option<Self>\n                where\n                    Self: Sized,\n                {\n                    use flat_serialize::FlatSerializable as _;\n                    if is_null {\n                        return None;\n                    }\n\n                    let mut ptr = pg_sys::pg_detoast_datum_packed(datum.cast_mut_ptr());\n                    //TODO is there a better way to do this?\n                    if pgrx::varatt_is_1b(ptr) {\n                        ptr = pg_sys::pg_detoast_datum_copy(ptr);\n                    }\n                    let data_len = pgrx::varsize_any(ptr);\n\n                    // NOTE: varlena types are aligned according to the `ALIGNMENT` with which they\n                    // are configured in CREATE TYPE. We have (historically) not configured the\n                    // `ALIGNMENT` of our types, resulting in them having the default (for varlena)\n                    // 4-byte alignment.\n                    // Some types can only be safely deserialized by `flat_serialize::try_ref` if\n                    // they are 8-byte aligned (structs which contain slices of 8-byte aligned data,\n                    // because of `flat_serialize::try_ref`'s usage of `slice::from_raw_parts`).\n                    //\n                    // To correct for this, when the data is not aligned, we copy it into a new,\n                    // aligned, memory location.\n                    // XXX: Technically, we're going to copy more frequently than strictly necessary\n                    // because `flat_serialize::try_ref` _can_ safely deserialize types which are\n                    // not 8-byte aligned, but contain fields which require 8-byte alignment (as\n                    // long as they're not slices) because it uses `ptr::read_unaligned`.\n                    let is_aligned = ptr.cast::<$name>().is_aligned();\n                    let bytes = if !is_aligned {\n                        let unaligned_bytes = std::slice::from_raw_parts(ptr as *mut u8, data_len);\n                        let new_bytes = pgrx::pg_sys::palloc0(data_len);\n\n                        // Note: we assume that fresh allocations are 8-byte aligned\n                        debug_assert!(new_bytes.cast::<$name>().is_aligned());\n\n                        let new_slice: &mut [u8] = std::slice::from_raw_parts_mut(new_bytes.cast(), data_len);\n                        new_slice.copy_from_slice(unaligned_bytes);\n                        new_slice\n                    } else {\n                        std::slice::from_raw_parts(ptr as *mut u8, data_len)\n                    };\n                    let (data, _) = match [<$name Data>]::try_ref(bytes) {\n                        Ok(wrapped) => wrapped,\n                        Err(e) => error!(concat!(\"invalid \", stringify!($name), \" {:?}, got len {}\"), e, bytes.len()),\n                    };\n\n                    $name(data, $crate::type_builder::CachedDatum::FromInput(bytes)).into()\n                }\n            }\n\n            impl<$lifetemplate> pgrx::IntoDatum for $name<$lifetemplate> {\n                fn into_datum(self) -> Option<pgrx::pg_sys::Datum> {\n                    use $crate::type_builder::CachedDatum::*;\n                    let datum = match self.1 {\n                        Flattened(bytes) => pg_sys::Datum::from(bytes.as_ptr()),\n                        FromInput(..) | None => pg_sys::Datum::from(self.0.to_pg_bytes().as_ptr()),\n                    };\n                    Some(datum)\n                }\n\n                fn type_oid() -> pg_sys::Oid {\n                    rust_regtypein::<Self>()\n                }\n            }\n\n            unsafe impl<$lifetemplate> ::pgrx::callconv::BoxRet for $name<$lifetemplate> {\n                unsafe fn box_into<'fcx>(\n                    self,\n                    fcinfo: &mut ::pgrx::callconv::FcInfo<'fcx>,\n                ) -> ::pgrx::datum::Datum<'fcx> {\n                    match ::pgrx::datum::IntoDatum::into_datum(self) {\n                        None => fcinfo.return_null(),\n                        Some(datum) => unsafe { fcinfo.return_raw_datum(datum) }\n                    }\n                }\n            }\n\n            unsafe impl<'fcx, $lifetemplate> callconv::ArgAbi<'fcx> for $name<$lifetemplate>\n            where\n                Self: 'fcx,\n            {\n                unsafe fn unbox_arg_unchecked(arg: callconv::Arg<'_, 'fcx>) -> Self {\n                    let index = arg.index();\n                    unsafe { arg.unbox_arg_using_from_datum().unwrap_or_else(|| panic!(\"argument {index} must not be null\")) }\n                }\n\n                unsafe fn unbox_nullable_arg(arg: callconv::Arg<'_, 'fcx>) -> nullable::Nullable<Self> {\n                    unsafe { arg.unbox_arg_using_from_datum().into() }\n                }\n            }\n\n            impl<$lifetemplate> ::std::ops::Deref for $name <$lifetemplate> {\n                type Target=[<$name Data>] $(<$inlife>)?;\n                fn deref(&self) -> &Self::Target {\n                    &self.0\n                }\n            }\n\n            impl<$lifetemplate> ::std::ops::DerefMut for $name <$lifetemplate> {\n                fn deref_mut(&mut self) -> &mut Self::Target {\n                    self.1 = $crate::type_builder::CachedDatum::None;\n                    &mut self.0\n                }\n            }\n\n            impl<$lifetemplate> From<[<$name Data>]$(<$inlife>)?> for $name<$lifetemplate> {\n                fn from(inner: [<$name Data>]$(<$inlife>)?) -> Self {\n                    Self(inner, $crate::type_builder::CachedDatum::None)\n                }\n            }\n\n            impl<$lifetemplate> From<[<$name Data>]$(<$inlife>)?> for Option<$name<$lifetemplate>> {\n                fn from(inner: [<$name Data>]$(<$inlife>)?) -> Self {\n                    Some($name(inner, $crate::type_builder::CachedDatum::None))\n                }\n            }\n        }\n    }\n}\n\n#[macro_export]\nmacro_rules! pg_type_no_lifetime_impl {\n    (\n        $(#[$attrs: meta])*\n        struct $name: ident {\n            $($(#[$fattrs: meta])* $field:ident : $typ: ty),*\n            $(,)?\n        }\n    ) => {\n        ::paste::paste! {\n            $(#[$attrs])*\n            #[derive(pgrx::PostgresType, Clone, serde::Serialize, serde::Deserialize)]\n            #[inoutfuncs]\n            #[bikeshed_postgres_type_manually_impl_from_into_datum]\n            pub struct $name(pub [<$name Data>], $crate::type_builder::CachedDatum<'static>);\n\n            flat_serialize_macro::flat_serialize! {\n                $(#[$attrs])*\n                #[derive(serde::Serialize, serde::Deserialize)]\n                struct [<$name Data>] {\n                    #[serde(skip, default=\"crate::serialization::serde_reference_adaptor::default_header\")]\n                    header: u32,\n                    version: u8,\n                    #[serde(skip, default=\"crate::serialization::serde_reference_adaptor::default_padding\")]\n                    padding: [u8; 3],\n                    $($(#[$fattrs])* $field: $typ),*\n                }\n            }\n\n            impl $name {\n                pub fn in_current_context(&self) -> $name {\n                    unsafe { self.0.flatten() }\n                }\n\n                #[allow(clippy::missing_safety_doc)]\n                pub unsafe fn cached_datum_or_flatten(&mut self) -> pgrx::pg_sys::Datum {\n                    use $crate::type_builder::CachedDatum::*;\n                    match self.1 {\n                        None => {\n                            *self = unsafe { self.0.flatten() };\n                            self.cached_datum_or_flatten()\n                        },\n                        FromInput(bytes) | Flattened(bytes) => pg_sys::Datum::from(bytes.as_ptr()),\n                    }\n                }\n            }\n\n            impl [<$name Data>] {\n                #[allow(clippy::missing_safety_doc)]\n                pub unsafe fn flatten(&self) -> $name {\n                    use flat_serialize::FlatSerializable as _;\n                    use $crate::type_builder::CachedDatum::Flattened;\n                    let bytes: &'static [u8] = self.to_pg_bytes();\n                    let wrapped = [<$name Data>]::try_ref(bytes).unwrap().0;\n                    $name(wrapped, Flattened(bytes))\n                }\n\n                pub fn to_pg_bytes(&self) -> &'static [u8] {\n                    use std::{mem::MaybeUninit, slice};\n                    use flat_serialize::FlatSerializable as _;\n                    unsafe {\n                        let len = self.num_bytes();\n                        // valena types have a maximum size\n                        if len > 0x3FFFFFFF {\n                            pgrx::error!(\"size {} bytes is to large\", len)\n                        }\n                        let memory: *mut MaybeUninit<u8> = pg_sys::palloc0(len).cast();\n                        let slice = slice::from_raw_parts_mut(memory, len);\n                        let rem = self.fill_slice(slice);\n                        debug_assert_eq!(rem.len(), 0);\n\n                        ::pgrx::set_varsize_4b(memory.cast(), len as i32);\n                        slice::from_raw_parts(memory.cast(), len)\n                    }\n                }\n            }\n\n            impl pgrx::FromDatum for $name {\n                unsafe fn from_polymorphic_datum(datum: pgrx::pg_sys::Datum, is_null: bool, _: pg_sys::Oid) -> Option<Self>\n                where\n                    Self: Sized,\n                {\n                    use flat_serialize::FlatSerializable as _;\n                    if is_null {\n                        return None;\n                    }\n\n                    let mut ptr = pg_sys::pg_detoast_datum_packed(datum.cast_mut_ptr());\n                    //TODO is there a better way to do this?\n                    if pgrx::varatt_is_1b(ptr) {\n                        ptr = pg_sys::pg_detoast_datum_copy(ptr);\n                    }\n                    let data_len = pgrx::varsize_any(ptr);\n\n                    let is_aligned = ptr.cast::<$name>().is_aligned();\n                    let bytes = if !is_aligned {\n                        let unaligned_bytes = std::slice::from_raw_parts(ptr as *mut u8, data_len);\n                        let new_bytes = pgrx::pg_sys::palloc0(data_len);\n\n                        debug_assert!(new_bytes.cast::<$name>().is_aligned());\n\n                        let new_slice: &mut [u8] = std::slice::from_raw_parts_mut(new_bytes.cast(), data_len);\n                        new_slice.copy_from_slice(unaligned_bytes);\n                        new_slice\n                    } else {\n                        std::slice::from_raw_parts(ptr as *mut u8, data_len)\n                    };\n                    let (data, _) = match [<$name Data>]::try_ref(bytes) {\n                        Ok(wrapped) => wrapped,\n                        Err(e) => error!(concat!(\"invalid \", stringify!($name), \" {:?}, got len {}\"), e, bytes.len()),\n                    };\n\n                    $name(data, $crate::type_builder::CachedDatum::FromInput(bytes)).into()\n                }\n            }\n\n            impl pgrx::IntoDatum for $name {\n                fn into_datum(self) -> Option<pgrx::pg_sys::Datum> {\n                    use $crate::type_builder::CachedDatum::*;\n                    let datum = match self.1 {\n                        Flattened(bytes) => pg_sys::Datum::from(bytes.as_ptr()),\n                        FromInput(..) | None => pg_sys::Datum::from(self.0.to_pg_bytes().as_ptr()),\n                    };\n                    Some(datum)\n                }\n\n                fn type_oid() -> pg_sys::Oid {\n                    rust_regtypein::<Self>()\n                }\n            }\n\n            unsafe impl ::pgrx::callconv::BoxRet for $name {\n                unsafe fn box_into<'fcx>(\n                    self,\n                    fcinfo: &mut ::pgrx::callconv::FcInfo<'fcx>,\n                ) -> ::pgrx::datum::Datum<'fcx> {\n                    match ::pgrx::datum::IntoDatum::into_datum(self) {\n                        None => fcinfo.return_null(),\n                        Some(datum) => unsafe { fcinfo.return_raw_datum(datum) }\n                    }\n                }\n            }\n\n            unsafe impl<'fcx> callconv::ArgAbi<'fcx> for $name\n            where\n                Self: 'fcx,\n            {\n                unsafe fn unbox_arg_unchecked(arg: callconv::Arg<'_, 'fcx>) -> Self {\n                    let index = arg.index();\n                    unsafe { arg.unbox_arg_using_from_datum().unwrap_or_else(|| panic!(\"argument {index} must not be null\")) }\n                }\n\n                unsafe fn unbox_nullable_arg(arg: callconv::Arg<'_, 'fcx>) -> nullable::Nullable<Self> {\n                    unsafe { arg.unbox_arg_using_from_datum().into() }\n                }\n            }\n\n            impl ::std::ops::Deref for $name {\n                type Target=[<$name Data>];\n                fn deref(&self) -> &Self::Target {\n                    &self.0\n                }\n            }\n\n            impl ::std::ops::DerefMut for $name {\n                fn deref_mut(&mut self) -> &mut Self::Target {\n                    self.1 = $crate::type_builder::CachedDatum::None;\n                    &mut self.0\n                }\n            }\n\n            impl From<[<$name Data>]> for $name {\n                fn from(inner: [<$name Data>]) -> Self {\n                    Self(inner, $crate::type_builder::CachedDatum::None)\n                }\n            }\n\n            impl From<[<$name Data>]> for Option<$name> {\n                fn from(inner: [<$name Data>]) -> Self {\n                    Some($name(inner, $crate::type_builder::CachedDatum::None))\n                }\n            }\n        }\n    }\n}\n\n// Routes to ron_inout_funcs_impl! for lifetime types_impl! for non-lifetime types\n#[macro_export]\nmacro_rules! ron_inout_funcs {\n    // Pattern 1: Explicit lifetime parameter → route to ron_inout_funcs_impl!\n    ($name:ident<$lifetime:lifetime>) => {\n        $crate::ron_inout_funcs_impl!($name);\n    };\n\n    // Pattern 2: No lifetime parameter → route to ron_inout_funcs_no_lifetime_impl!\n    ($name:ident) => {\n        $crate::ron_inout_funcs_no_lifetime_impl!($name);\n    };\n}\n\n// Implementation macro for types without lifetimes (used internally by unified ron_inout_funcs!)\n#[macro_export]\nmacro_rules! ron_inout_funcs_no_lifetime_impl {\n    ($name:ident) => {\n        impl InOutFuncs for $name {\n            fn output(&self, buffer: &mut StringInfo) {\n                use $crate::serialization::{str_to_db_encoding, EncodedStr::*};\n\n                let stringified = ron::to_string(&**self).unwrap();\n                match str_to_db_encoding(&stringified) {\n                    Utf8(s) => buffer.push_str(s),\n                    Other(s) => buffer.push_bytes(s.to_bytes()),\n                }\n            }\n\n            fn input(input: &std::ffi::CStr) -> $name\n            where\n                Self: Sized,\n            {\n                use $crate::serialization::str_from_db_encoding;\n\n                let input = str_from_db_encoding(input);\n                let val = ron::from_str(input).unwrap();\n                unsafe { Self(val, $crate::type_builder::CachedDatum::None).flatten() }\n            }\n        }\n    };\n}\n\n// Implementation macro for lifetime-parameterized types (used internally by unified ron_inout_funcs!)\n#[macro_export]\nmacro_rules! ron_inout_funcs_impl {\n    ($name:ident) => {\n        impl<'input> InOutFuncs for $name<'input> {\n            fn output(&self, buffer: &mut StringInfo) {\n                use $crate::serialization::{str_to_db_encoding, EncodedStr::*};\n\n                let stringified = ron::to_string(&**self).unwrap();\n                match str_to_db_encoding(&stringified) {\n                    Utf8(s) => buffer.push_str(s),\n                    Other(s) => buffer.push_bytes(s.to_bytes()),\n                }\n            }\n\n            fn input(input: &std::ffi::CStr) -> $name<'input>\n            where\n                Self: Sized,\n            {\n                use $crate::serialization::str_from_db_encoding;\n\n                let input = str_from_db_encoding(input);\n                let val = ron::from_str(input).unwrap();\n                unsafe { Self(val, $crate::type_builder::CachedDatum::None).flatten() }\n            }\n        }\n    };\n}\n\n#[macro_export]\nmacro_rules! flatten {\n    ($typ:ident { $($field:ident$(: $value:expr)?),* $(,)? }) => {\n        {\n            let data = ::paste::paste! {\n                [<$typ Data>] {\n                    header: 0,\n                    version: 1,\n                    padding: [0; 3],\n                    $(\n                        $field$(: $value)?\n                    ),*\n                }\n            };\n            data.flatten()\n        }\n    }\n}\n\n#[macro_export]\nmacro_rules! build {\n    ($typ:ident { $($field:ident$(: $value:expr)?),* $(,)? }) => {\n        {\n            <$typ>::from(::paste::paste! {\n                [<$typ Data>] {\n                    header: 0,\n                    version: 1,\n                    padding: [0; 3],\n                    $(\n                        $field$(: $value)?\n                    ),*\n                }\n            })\n        }\n    }\n}\n\n#[repr(u8)]\npub enum SerializationType {\n    Default = 1,\n}\n\n#[macro_export]\nmacro_rules! do_serialize {\n    ($state: ident) => {\n        {\n            $crate::do_serialize!($state, version: 1)\n        }\n    };\n    ($state: ident, version: $version: expr) => {\n        {\n            use $crate::type_builder::SerializationType;\n            use std::io::{Cursor, Write};\n            use std::convert::TryInto;\n\n            let state = &*$state;\n            let serialized_size = bincode::serialized_size(state)\n                .unwrap_or_else(|e| pgrx::error!(\"serialization error {}\", e));\n            let our_size = serialized_size + 2; // size of serialized data + our version flags\n            let allocated_size = our_size + 4; // size of our data + the varlena header\n            let allocated_size = allocated_size.try_into()\n                .unwrap_or_else(|e| pgrx::error!(\"serialization error {}\", e));\n            // valena types have a maximum size\n            if allocated_size > 0x3FFFFFFF {\n                pgrx::error!(\"size {} bytes is to large\", allocated_size)\n            }\n\n            let bytes: &mut [u8] = unsafe {\n                let bytes = pgrx::pg_sys::palloc0(allocated_size);\n                std::slice::from_raw_parts_mut(bytes.cast(), allocated_size)\n            };\n            let mut writer = Cursor::new(bytes);\n            // varlena header space\n            let varsize = [0; 4];\n            writer.write_all(&varsize)\n                .unwrap_or_else(|e| pgrx::error!(\"serialization error {}\", e));\n            // type version\n            writer.write_all(&[$version])\n                .unwrap_or_else(|e| pgrx::error!(\"serialization error {}\", e));\n            // serialization version; 1 for bincode is currently the only option\n            writer.write_all(&[SerializationType::Default as u8])\n                .unwrap_or_else(|e| pgrx::error!(\"serialization error {}\", e));\n            bincode::serialize_into(&mut writer, state)\n                .unwrap_or_else(|e| pgrx::error!(\"serialization error {}\", e));\n            unsafe {\n                let len = writer.position().try_into().expect(\"serialized size too large\");\n                ::pgrx::set_varsize_4b(writer.get_mut().as_mut_ptr() as *mut _, len);\n            }\n            $crate::raw::bytea::from(pg_sys::Datum::from(writer.into_inner().as_mut_ptr()))\n        }\n    };\n}\n\n#[macro_export]\nmacro_rules! do_deserialize {\n    ($bytes: expr, $t: ty) => {{\n        use $crate::type_builder::SerializationType;\n\n        let input: $crate::raw::bytea = $bytes;\n        let state: $t = unsafe {\n            let input: pgrx::pg_sys::Datum = input.into();\n            let detoasted = pg_sys::pg_detoast_datum_packed(input.cast_mut_ptr());\n            let len = pgrx::varsize_any_exhdr(detoasted);\n            let data = pgrx::vardata_any(detoasted);\n            let bytes = std::slice::from_raw_parts(data as *mut u8, len);\n            if bytes.len() < 1 {\n                pgrx::error!(\"deserialization error, no bytes\")\n            }\n            if bytes[0] != 1 {\n                pgrx::error!(\n                    \"deserialization error, invalid serialization version {}\",\n                    bytes[0]\n                )\n            }\n            if bytes[1] != SerializationType::Default as u8 {\n                pgrx::error!(\n                    \"deserialization error, invalid serialization type {}\",\n                    bytes[1]\n                )\n            }\n            bincode::deserialize(&bytes[2..])\n                .unwrap_or_else(|e| pgrx::error!(\"deserialization error {}\", e))\n        };\n        state.into()\n    }};\n}\n"
  },
  {
    "path": "extension/src/uddsketch.rs",
    "content": "use pgrx::*;\n\nuse encodings::{delta, prefix_varint};\n\nuse uddsketch::{SketchHashKey, UDDSketch as UddSketchInternal, UDDSketchMetadata};\n\nuse crate::{\n    accessors::{\n        AccessorApproxPercentile, AccessorApproxPercentileRank, AccessorError, AccessorMean,\n        AccessorNumVals, AccessorPercentileArray,\n    },\n    aggregate_utils::in_aggregate_context,\n    flatten,\n    palloc::{Inner, Internal, InternalAsValue, ToInternal},\n    pg_type,\n};\n\n// PG function for adding values to a sketch.\n// Null values are ignored.\n#[pg_extern(immutable, parallel_safe)]\npub fn uddsketch_trans(\n    state: Internal,\n    size: i32,\n    max_error: f64,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    uddsketch_trans_inner(unsafe { state.to_inner() }, size, max_error, value, fcinfo).internal()\n}\n\npub fn uddsketch_trans_inner(\n    state: Option<Inner<UddSketchInternal>>,\n    size: i32,\n    max_error: f64,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<UddSketchInternal>> {\n    let max_size = u32::try_from(size).unwrap_or(PERCENTILE_AGG_DEFAULT_SIZE);\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let value = match value {\n                None => return state,\n                Some(value) => value,\n            };\n            let mut state = match state {\n                None => UddSketchInternal::new(max_size, max_error).into(),\n                Some(state) => state,\n            };\n            state.add_value(value);\n            Some(state)\n        })\n    }\n}\n\nconst PERCENTILE_AGG_DEFAULT_SIZE: u32 = 200;\nconst PERCENTILE_AGG_DEFAULT_ERROR: f64 = 0.001;\n\n// transition function for the simpler percentile_agg aggregate, which doesn't\n// take parameters for the size and error, but uses a default\n#[pg_extern(immutable, parallel_safe)]\npub fn percentile_agg_trans(\n    state: Internal,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    percentile_agg_trans_inner(unsafe { state.to_inner() }, value, fcinfo).internal()\n}\n\npub fn percentile_agg_trans_inner(\n    state: Option<Inner<UddSketchInternal>>,\n    value: Option<f64>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<UddSketchInternal>> {\n    let default_size = PERCENTILE_AGG_DEFAULT_SIZE;\n    let default_max_error = PERCENTILE_AGG_DEFAULT_ERROR;\n    uddsketch_trans_inner(state, default_size as _, default_max_error, value, fcinfo)\n}\n// PG function for merging sketches.\n#[pg_extern(immutable, parallel_safe)]\npub fn uddsketch_combine(\n    state1: Internal,\n    state2: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { uddsketch_combine_inner(state1.to_inner(), state2.to_inner(), fcinfo).internal() }\n}\npub fn uddsketch_combine_inner(\n    state1: Option<Inner<UddSketchInternal>>,\n    state2: Option<Inner<UddSketchInternal>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<UddSketchInternal>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || match (state1, state2) {\n            (None, None) => None,\n            (None, Some(state2)) => Some(state2.clone().into()),\n            (Some(state1), None) => Some(state1.clone().into()),\n            (Some(state1), Some(state2)) => {\n                let mut sketch = state1.clone();\n                sketch.merge_sketch(&state2);\n                Some(sketch.into())\n            }\n        })\n    }\n}\n\nuse crate::raw::bytea;\n\n#[pg_extern(immutable, parallel_safe, strict)]\npub fn uddsketch_serialize(state: Internal) -> bytea {\n    let serializable = &SerializedUddSketch::from(unsafe { state.get().unwrap() });\n    crate::do_serialize!(serializable)\n}\n\n#[pg_extern(strict, immutable, parallel_safe)]\npub fn uddsketch_deserialize(bytes: bytea, _internal: Internal) -> Option<Internal> {\n    uddsketch_deserialize_inner(bytes).internal()\n}\npub fn uddsketch_deserialize_inner(bytes: bytea) -> Inner<UddSketchInternal> {\n    let sketch: UddSketchInternal = crate::do_deserialize!(bytes, SerializedUddSketch);\n    sketch.into()\n}\n\n#[derive(serde::Serialize, serde::Deserialize)]\nstruct SerializedUddSketch {\n    alpha: f64,\n    max_buckets: u32,\n    num_buckets: u32,\n    compactions: u32,\n    count: u64,\n    sum: f64,\n    buckets: CompressedBuckets,\n}\n\nimpl From<&UddSketchInternal> for SerializedUddSketch {\n    fn from(sketch: &UddSketchInternal) -> Self {\n        let buckets = compress_buckets(sketch.bucket_iter());\n        SerializedUddSketch {\n            alpha: sketch.max_error(),\n            max_buckets: sketch.max_allowed_buckets(),\n            num_buckets: sketch.current_buckets_count() as u32,\n            compactions: u32::from(sketch.times_compacted()),\n            count: sketch.count(),\n            sum: sketch.sum(),\n            buckets,\n        }\n    }\n}\n\nimpl From<SerializedUddSketch> for UddSketchInternal {\n    fn from(sketch: SerializedUddSketch) -> Self {\n        UddSketchInternal::new_from_data(\n            &UDDSketchMetadata {\n                max_buckets: sketch.max_buckets,\n                current_error: sketch.alpha,\n                compactions: u8::try_from(sketch.compactions)\n                    .expect(\"compactions cannot be higher than 65\"),\n                values: sketch.count,\n                sum: sketch.sum,\n                buckets: sketch.num_buckets,\n            },\n            sketch.keys(),\n            sketch.counts(),\n        )\n    }\n}\n\nimpl SerializedUddSketch {\n    fn keys(&self) -> impl Iterator<Item = SketchHashKey> + '_ {\n        decompress_keys(\n            &self.buckets.negative_indexes,\n            self.buckets.zero_bucket_count != 0,\n            &self.buckets.positive_indexes,\n        )\n    }\n\n    fn counts(&self) -> impl Iterator<Item = u64> + '_ {\n        decompress_counts(\n            &self.buckets.negative_counts,\n            self.buckets.zero_bucket_count,\n            &self.buckets.positive_counts,\n        )\n    }\n}\n\n// PG object for the sketch.\npg_type! {\n    #[derive(Debug)]\n    struct UddSketch<'input> {\n        alpha: f64,\n        max_buckets: u32,\n        num_buckets: u32,\n        compactions: u64,\n        count: u64,\n        sum: f64,\n        zero_bucket_count: u64,\n        neg_indexes_bytes: u32,\n        neg_buckets_bytes: u32,\n        pos_indexes_bytes: u32,\n        pos_buckets_bytes: u32,\n        negative_indexes: [u8; self.neg_indexes_bytes],\n        negative_counts: [u8; self.neg_buckets_bytes],\n        positive_indexes: [u8; self.pos_indexes_bytes],\n        positive_counts: [u8; self.pos_buckets_bytes],\n    }\n}\n\n#[derive(serde::Serialize, serde::Deserialize)]\nstruct ReadableUddSketch {\n    version: u8,\n    alpha: f64,\n    max_buckets: u32,\n    num_buckets: u32,\n    compactions: u64,\n    count: u64,\n    sum: f64,\n    buckets: Vec<(SketchHashKey, u64)>,\n}\n\nimpl From<&UddSketch<'_>> for ReadableUddSketch {\n    fn from(sketch: &UddSketch<'_>) -> Self {\n        ReadableUddSketch {\n            version: sketch.version,\n            alpha: sketch.alpha,\n            max_buckets: sketch.max_buckets,\n            num_buckets: sketch.num_buckets,\n            compactions: sketch.compactions,\n            count: sketch.count,\n            sum: sketch.sum,\n            buckets: sketch.keys().zip(sketch.counts()).collect(),\n        }\n    }\n}\n\nimpl<'a, 'b> From<&'a ReadableUddSketch> for UddSketch<'b> {\n    fn from(sketch: &'a ReadableUddSketch) -> Self {\n        assert_eq!(sketch.version, 1);\n\n        let CompressedBuckets {\n            negative_indexes,\n            negative_counts,\n            zero_bucket_count,\n            positive_indexes,\n            positive_counts,\n        } = compress_buckets(sketch.buckets.iter().cloned());\n\n        unsafe {\n            flatten! {\n                UddSketch {\n                    alpha: sketch.alpha,\n                    max_buckets: sketch.max_buckets,\n                    num_buckets: sketch.num_buckets,\n                    compactions: sketch.compactions,\n                    count: sketch.count,\n                    sum: sketch.sum,\n                    zero_bucket_count,\n                    neg_indexes_bytes: (negative_indexes.len() as u32),\n                    neg_buckets_bytes: (negative_counts.len() as u32),\n                    pos_indexes_bytes: (positive_indexes.len() as u32),\n                    pos_buckets_bytes: (positive_counts.len() as u32),\n                    negative_indexes: (&*negative_indexes).into(),\n                    negative_counts: (&*negative_counts).into(),\n                    positive_indexes: (&*positive_indexes).into(),\n                    positive_counts: (&*positive_counts).into(),\n                }\n            }\n        }\n    }\n}\n\nimpl<'input> InOutFuncs for UddSketch<'input> {\n    fn output(&self, buffer: &mut StringInfo) {\n        use crate::serialization::{str_to_db_encoding, EncodedStr::*};\n\n        let stringified = ron::to_string(&ReadableUddSketch::from(self)).unwrap();\n        match str_to_db_encoding(&stringified) {\n            Utf8(s) => buffer.push_str(s),\n            Other(s) => buffer.push_bytes(s.to_bytes()),\n        }\n    }\n\n    fn input(input: &std::ffi::CStr) -> Self\n    where\n        Self: Sized,\n    {\n        use crate::serialization::str_from_db_encoding;\n\n        let utf8_str = str_from_db_encoding(input);\n        let val: ReadableUddSketch = ron::from_str(utf8_str).unwrap();\n        UddSketch::from(&val)\n    }\n}\n\nimpl<'input> UddSketch<'input> {\n    fn keys(&self) -> impl Iterator<Item = SketchHashKey> + '_ {\n        // FIXME does this really need a slice?\n        decompress_keys(\n            self.negative_indexes.as_slice(),\n            self.zero_bucket_count != 0,\n            self.positive_indexes.as_slice(),\n        )\n    }\n\n    fn counts(&self) -> impl Iterator<Item = u64> + '_ {\n        // FIXME does this really need a slice?\n        decompress_counts(\n            self.negative_counts.as_slice(),\n            self.zero_bucket_count,\n            self.positive_counts.as_slice(),\n        )\n    }\n\n    fn metadata(&self) -> UDDSketchMetadata {\n        UDDSketchMetadata {\n            max_buckets: self.max_buckets,\n            current_error: self.alpha,\n            compactions: u8::try_from(self.compactions)\n                .expect(\"compactions cannot be higher than 65\"),\n            values: self.count,\n            sum: self.sum,\n            buckets: self.num_buckets,\n        }\n    }\n\n    fn to_uddsketch(&self) -> UddSketchInternal {\n        UddSketchInternal::new_from_data(&self.metadata(), self.keys(), self.counts())\n    }\n\n    fn from_internal(state: &UddSketchInternal) -> Self {\n        let CompressedBuckets {\n            negative_indexes,\n            negative_counts,\n            zero_bucket_count,\n            positive_indexes,\n            positive_counts,\n        } = compress_buckets(state.bucket_iter());\n\n        // we need to flatten the vector to a single buffer that contains\n        // both the size, the data, and the varlen header\n        unsafe {\n            flatten!(UddSketch {\n                alpha: state.max_error(),\n                max_buckets: state.max_allowed_buckets(),\n                num_buckets: state.current_buckets_count() as u32,\n                compactions: state.times_compacted() as u64,\n                count: state.count(),\n                sum: state.sum(),\n                zero_bucket_count,\n                neg_indexes_bytes: negative_indexes.len() as u32,\n                neg_buckets_bytes: negative_counts.len() as u32,\n                pos_indexes_bytes: positive_indexes.len() as u32,\n                pos_buckets_bytes: positive_counts.len() as u32,\n                negative_indexes: negative_indexes.into(),\n                negative_counts: negative_counts.into(),\n                positive_indexes: positive_indexes.into(),\n                positive_counts: positive_counts.into(),\n            })\n        }\n    }\n}\n\nimpl<'input> FromIterator<f64> for UddSketch<'input> {\n    fn from_iter<T: IntoIterator<Item = f64>>(iter: T) -> Self {\n        let mut sketch = UddSketchInternal::new(\n            PERCENTILE_AGG_DEFAULT_SIZE.into(),\n            PERCENTILE_AGG_DEFAULT_ERROR,\n        );\n        for value in iter {\n            sketch.add_value(value);\n        }\n        Self::from_internal(&sketch)\n    }\n}\n\n// PG function to generate a user-facing UddSketch object from a UddSketchInternal.\n#[pg_extern(immutable, parallel_safe)]\nfn uddsketch_final(\n    state: Internal,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<UddSketch<'static>> {\n    unsafe { uddsketch_final_inner(state.to_inner(), fcinfo) }\n}\nfn uddsketch_final_inner(\n    state: Option<Inner<UddSketchInternal>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<UddSketch<'static>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let state = match state {\n                None => return None,\n                Some(state) => state,\n            };\n\n            UddSketch::from_internal(&state).into()\n        })\n    }\n}\n\n#[derive(serde::Serialize, serde::Deserialize)]\nstruct CompressedBuckets {\n    negative_indexes: Vec<u8>,\n    negative_counts: Vec<u8>,\n    zero_bucket_count: u64,\n    positive_indexes: Vec<u8>,\n    positive_counts: Vec<u8>,\n}\n\nfn compress_buckets(buckets: impl Iterator<Item = (SketchHashKey, u64)>) -> CompressedBuckets {\n    let mut negative_indexes = prefix_varint::I64Compressor::with(delta::i64_encoder());\n    let mut negative_counts = prefix_varint::U64Compressor::with(delta::u64_encoder());\n    let mut zero_bucket_count = 0;\n    let mut positive_indexes = prefix_varint::I64Compressor::with(delta::i64_encoder());\n    let mut positive_counts = prefix_varint::U64Compressor::with(delta::u64_encoder());\n    for (k, b) in buckets {\n        match k {\n            SketchHashKey::Negative(i) => {\n                negative_indexes.push(i);\n                negative_counts.push(b);\n            }\n            SketchHashKey::Zero => zero_bucket_count = b,\n            SketchHashKey::Positive(i) => {\n                positive_indexes.push(i);\n                positive_counts.push(b);\n            }\n            SketchHashKey::Invalid => unreachable!(),\n        }\n    }\n    let negative_indexes = negative_indexes.finish();\n    let negative_counts = negative_counts.finish();\n    let positive_indexes = positive_indexes.finish();\n    let positive_counts = positive_counts.finish();\n    CompressedBuckets {\n        negative_indexes,\n        negative_counts,\n        zero_bucket_count,\n        positive_indexes,\n        positive_counts,\n    }\n}\n\nfn decompress_keys<'i>(\n    negative_indexes: &'i [u8],\n    zero_bucket: bool,\n    positive_indexes: &'i [u8],\n) -> impl Iterator<Item = SketchHashKey> + 'i {\n    let negatives = prefix_varint::i64_decompressor(negative_indexes)\n        .map(delta::i64_decoder())\n        .map(SketchHashKey::Negative);\n\n    let zero = zero_bucket.then(|| uddsketch::SketchHashKey::Zero);\n\n    let positives = prefix_varint::i64_decompressor(positive_indexes)\n        .map(delta::i64_decoder())\n        .map(SketchHashKey::Positive);\n\n    negatives.chain(zero).chain(positives)\n}\n\nfn decompress_counts<'b>(\n    negative_buckets: &'b [u8],\n    zero_bucket: u64,\n    positive_buckets: &'b [u8],\n) -> impl Iterator<Item = u64> + 'b {\n    let negatives = prefix_varint::u64_decompressor(negative_buckets).map(delta::u64_decoder());\n    let zero = (zero_bucket != 0).then(|| zero_bucket);\n    let positives = prefix_varint::u64_decompressor(positive_buckets).map(delta::u64_decoder());\n\n    negatives.chain(zero).chain(positives)\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE uddsketch(\\n\\\n        size integer, max_error DOUBLE PRECISION, value DOUBLE PRECISION\\n\\\n    ) (\\n\\\n        sfunc = uddsketch_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = uddsketch_final,\\n\\\n        combinefunc = uddsketch_combine,\\n\\\n        serialfunc = uddsketch_serialize,\\n\\\n        deserialfunc = uddsketch_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"udd_agg\",\n    requires = [\n        uddsketch_trans,\n        uddsketch_final,\n        uddsketch_combine,\n        uddsketch_serialize,\n        uddsketch_deserialize\n    ],\n);\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE percentile_agg(value DOUBLE PRECISION)\\n\\\n    (\\n\\\n        sfunc = percentile_agg_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = uddsketch_final,\\n\\\n        combinefunc = uddsketch_combine,\\n\\\n        serialfunc = uddsketch_serialize,\\n\\\n        deserialfunc = uddsketch_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"percentile_agg\",\n    requires = [\n        percentile_agg_trans,\n        uddsketch_final,\n        uddsketch_combine,\n        uddsketch_serialize,\n        uddsketch_deserialize\n    ],\n);\n\n#[pg_extern(immutable, parallel_safe)]\npub fn uddsketch_compound_trans<'a>(\n    state: Internal,\n    value: Option<UddSketch<'a>>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Internal> {\n    unsafe { uddsketch_compound_trans_inner(state.to_inner(), value, fcinfo).internal() }\n}\npub fn uddsketch_compound_trans_inner(\n    state: Option<Inner<UddSketchInternal>>,\n    value: Option<UddSketch>,\n    fcinfo: pg_sys::FunctionCallInfo,\n) -> Option<Inner<UddSketchInternal>> {\n    unsafe {\n        in_aggregate_context(fcinfo, || {\n            let Some(value) = value else { return state };\n            let Some(mut state) = state else {\n                return Some(value.to_uddsketch().into());\n            };\n\n            state.merge_items(&value.metadata(), value.keys(), value.counts());\n            state.into()\n        })\n    }\n}\n\nextension_sql!(\n    \"\\n\\\n    CREATE AGGREGATE rollup(\\n\\\n        sketch uddsketch\\n\\\n    ) (\\n\\\n        sfunc = uddsketch_compound_trans,\\n\\\n        stype = internal,\\n\\\n        finalfunc = uddsketch_final,\\n\\\n        combinefunc = uddsketch_combine,\\n\\\n        serialfunc = uddsketch_serialize,\\n\\\n        deserialfunc = uddsketch_deserialize,\\n\\\n        parallel = safe\\n\\\n    );\\n\\\n\",\n    name = \"udd_rollup\",\n    requires = [\n        uddsketch_compound_trans,\n        uddsketch_final,\n        uddsketch_combine,\n        uddsketch_serialize,\n        uddsketch_deserialize\n    ],\n);\n\n//---- Available PG operations on the sketch\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_uddsketch_approx_percentile<'a>(\n    sketch: UddSketch<'a>,\n    accessor: AccessorApproxPercentile,\n) -> f64 {\n    uddsketch_approx_percentile(accessor.percentile, sketch)\n}\n\n// Approximate the value at the given approx_percentile (0.0-1.0)\n#[pg_extern(immutable, parallel_safe, name = \"approx_percentile\")]\npub fn uddsketch_approx_percentile<'a>(percentile: f64, sketch: UddSketch<'a>) -> f64 {\n    uddsketch::estimate_quantile(\n        percentile,\n        sketch.alpha,\n        uddsketch::gamma(sketch.alpha),\n        sketch.count,\n        sketch.keys().zip(sketch.counts()),\n    )\n}\n\n#[pg_operator(immutable)]\n#[opname(->)]\npub fn arrow_uddsketch_approx_percentile_array<'a>(\n    sketch: UddSketch<'a>,\n    percentiles: AccessorPercentileArray,\n) -> Vec<f64> {\n    approx_percentile_slice(&percentiles.percentile[..percentiles.len as usize], sketch)\n}\n\n// Approximate the value at the given approx_percentile (0.0-1.0) for each entry in an array\n#[pg_extern(immutable, name = \"approx_percentile_array\")]\npub fn uddsketch_approx_percentile_array<'a>(\n    percentiles: Vec<f64>,\n    sketch: UddSketch<'a>,\n) -> Vec<f64> {\n    approx_percentile_slice(&percentiles, sketch)\n}\n\nfn approx_percentile_slice<'a, 'b>(\n    percentiles: impl IntoIterator<Item = &'b f64>,\n    sketch: UddSketch<'a>,\n) -> Vec<f64> {\n    let mut results = Vec::new();\n    for percentile in percentiles {\n        results.push(uddsketch::estimate_quantile(\n            *percentile,\n            sketch.alpha,\n            uddsketch::gamma(sketch.alpha),\n            sketch.count,\n            sketch.keys().zip(sketch.counts()),\n        ))\n    }\n    results\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_uddsketch_approx_rank<'a>(\n    sketch: UddSketch<'a>,\n    accessor: AccessorApproxPercentileRank,\n) -> f64 {\n    uddsketch_approx_percentile_rank(accessor.value, sketch)\n}\n\n// Approximate the approx_percentile at the given value\n#[pg_extern(immutable, parallel_safe, name = \"approx_percentile_rank\")]\npub fn uddsketch_approx_percentile_rank<'a>(value: f64, sketch: UddSketch<'a>) -> f64 {\n    uddsketch::estimate_quantile_at_value(\n        value,\n        uddsketch::gamma(sketch.alpha),\n        sketch.count,\n        sketch.keys().zip(sketch.counts()),\n    )\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_uddsketch_num_vals<'a>(sketch: UddSketch<'a>, _accessor: AccessorNumVals) -> f64 {\n    uddsketch_num_vals(sketch)\n}\n\n// Number of elements from which the sketch was built.\n#[pg_extern(immutable, parallel_safe, name = \"num_vals\")]\npub fn uddsketch_num_vals<'a>(sketch: UddSketch<'a>) -> f64 {\n    sketch.count as f64\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_uddsketch_mean<'a>(sketch: UddSketch<'a>, _accessor: AccessorMean) -> f64 {\n    uddsketch_mean(sketch)\n}\n\n// Average of all the values entered in the sketch.\n// Note that this is not an approximation, though there may be loss of precision.\n#[pg_extern(immutable, parallel_safe, name = \"mean\")]\npub fn uddsketch_mean<'a>(sketch: UddSketch<'a>) -> f64 {\n    if sketch.count > 0 {\n        sketch.sum / sketch.count as f64\n    } else {\n        0.0\n    }\n}\n\n// Total sum of all the values entered in the sketch.\n#[pg_extern(immutable, parallel_safe, name = \"total\")]\npub fn uddsketch_sum(sketch: UddSketch<'_>) -> f64 {\n    sketch.sum\n}\n\n#[pg_operator(immutable, parallel_safe)]\n#[opname(->)]\npub fn arrow_uddsketch_error<'a>(sketch: UddSketch<'a>, _accessor: AccessorError) -> f64 {\n    uddsketch_error(sketch)\n}\n\n// The maximum error (relative to the true value) for any approx_percentile estimate.\n#[pg_extern(immutable, parallel_safe, name = \"error\")]\npub fn uddsketch_error<'a>(sketch: UddSketch<'a>) -> f64 {\n    sketch.alpha\n}\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use super::*;\n\n    use pgrx_macros::pg_test;\n\n    // Assert equality between two floats, within some fixed error range.\n    fn apx_eql(value: f64, expected: f64, error: f64) {\n        assert!(\n            (value - expected).abs() < error,\n            \"Float value {value} differs from expected {expected} by more than {error}\"\n        );\n    }\n\n    // Assert equality between two floats, within an error expressed as a fraction of the expected value.\n    fn pct_eql(value: f64, expected: f64, pct_error: f64) {\n        apx_eql(value, expected, pct_error * expected);\n    }\n\n    #[pg_test]\n    fn test_aggregate() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE test (data DOUBLE PRECISION)\", None, &[])\n                .unwrap();\n            client\n                .update(\n                    \"INSERT INTO test SELECT generate_series(0.01, 100, 0.01)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(10000), sanity);\n\n            client\n                .update(\n                    \"CREATE VIEW sketch AS \\\n                SELECT uddsketch(100, 0.05, data) \\\n                FROM test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM sketch\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert!(sanity.unwrap_or(0) > 0);\n\n            let (mean, count) = client\n                .update(\n                    \"SELECT \\\n                    mean(uddsketch), \\\n                    num_vals(uddsketch) \\\n                    FROM sketch\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            apx_eql(mean.unwrap(), 50.005, 0.0001);\n            apx_eql(count.unwrap(), 10000.0, 0.000001);\n\n            let (mean2, count2) = client\n                .update(\n                    \"SELECT \\\n                    uddsketch -> mean(), \\\n                    uddsketch -> num_vals() \\\n                    FROM sketch\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n            assert_eq!(mean, mean2);\n            assert_eq!(count, count2);\n\n            let (error, error2) = client\n                .update(\n                    \"SELECT \\\n                    error(uddsketch), \\\n                    uddsketch -> error() \\\n                    FROM sketch\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            apx_eql(error.unwrap(), 0.05, 0.0001);\n            assert_eq!(error, error2);\n\n            for i in 0..=100 {\n                let value = i as f64;\n                let approx_percentile = value / 100.0;\n\n                let (est_val, est_quant) = client\n                    .update(\n                        &format!(\n                            \"SELECT \\\n                                approx_percentile({approx_percentile}, uddsketch), \\\n                                approx_percentile_rank({value}, uddsketch) \\\n                            FROM sketch\"\n                        ),\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_two::<f64, f64>()\n                    .unwrap();\n\n                if i == 0 {\n                    pct_eql(est_val.unwrap(), 0.01, 1.0);\n                    apx_eql(est_quant.unwrap(), approx_percentile, 0.0001);\n                } else {\n                    pct_eql(est_val.unwrap(), value, 1.0);\n                    pct_eql(est_quant.unwrap(), approx_percentile, 1.0);\n                }\n\n                let (est_val2, est_quant2) = client\n                    .update(\n                        &format!(\n                            \"SELECT \\\n                                uddsketch->approx_percentile({approx_percentile}), \\\n                                uddsketch->approx_percentile_rank({value}) \\\n                            FROM sketch\"\n                        ),\n                        None,\n                        &[],\n                    )\n                    .unwrap()\n                    .first()\n                    .get_two::<f64, f64>()\n                    .unwrap();\n                assert_eq!(est_val, est_val2);\n                assert_eq!(est_quant, est_quant2);\n            }\n        });\n    }\n\n    #[pg_test]\n    fn test_compound_agg() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE new_test (device INTEGER, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"INSERT INTO new_test SELECT dev, dev - v FROM generate_series(1,10) dev, generate_series(0, 1.0, 0.01) v\", None, &[]).unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM new_test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(1010), sanity);\n\n            client\n                .update(\n                    \"CREATE VIEW sketches AS \\\n                SELECT device, uddsketch(20, 0.01, value) \\\n                FROM new_test \\\n                GROUP BY device\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW composite AS \\\n                SELECT rollup(uddsketch) as uddsketch \\\n                FROM sketches\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW base AS \\\n                SELECT uddsketch(20, 0.01, value) \\\n                FROM new_test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (value, error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile(0.9, uddsketch), \\\n                    error(uddsketch) \\\n                    FROM base\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            let (test_value, test_error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile(0.9, uddsketch), \\\n                    error(uddsketch) \\\n                    FROM composite\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            apx_eql(test_value.unwrap(), value.unwrap(), 0.0001);\n            apx_eql(test_error.unwrap(), error.unwrap(), 0.000001);\n            pct_eql(test_value.unwrap(), 9.0, test_error.unwrap());\n        });\n    }\n\n    #[pg_test]\n    fn test_percentile_agg() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE pa_test (device INTEGER, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"INSERT INTO pa_test SELECT dev, dev - v FROM generate_series(1,10) dev, generate_series(0, 1.0, 0.01) v\", None, &[]).unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM pa_test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(1010), sanity);\n\n            // use the default values for percentile_agg\n            client\n                .update(\n                    \"CREATE VIEW uddsketch_test AS \\\n                SELECT uddsketch(200, 0.001, value) as approx \\\n                FROM pa_test \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW percentile_agg AS \\\n                SELECT percentile_agg(value) as approx \\\n                FROM pa_test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (value, error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile(0.9, approx), \\\n                    error(approx) \\\n                    FROM uddsketch_test\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            let (test_value, test_error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile(0.9, approx), \\\n                    error(approx) \\\n                    FROM percentile_agg\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<f64, f64>()\n                .unwrap();\n\n            apx_eql(test_value.unwrap(), value.unwrap(), 0.0001);\n            apx_eql(test_error.unwrap(), error.unwrap(), 0.000001);\n            pct_eql(test_value.unwrap(), 9.0, test_error.unwrap());\n        });\n    }\n    #[pg_test]\n    fn test_approx_percentile_array() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE paa_test (device INTEGER, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"INSERT INTO paa_test SELECT dev, dev - v FROM generate_series(1,10) dev, generate_series(0, 1.0, 0.01) v\", None, &[]).unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM paa_test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(1010), sanity);\n\n            client\n                .update(\n                    \"CREATE VIEW uddsketch_test AS \\\n                SELECT uddsketch(200, 0.001, value) as approx \\\n                FROM paa_test \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW percentile_agg AS \\\n                SELECT percentile_agg(value) as approx \\\n                FROM paa_test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (value, error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile_array(array[0.9,0.5,0.2], approx), \\\n                    error(approx) \\\n                    FROM uddsketch_test\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<Vec<f64>, f64>()\n                .unwrap();\n\n            let (test_value, test_error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile_array(array[0.9,0.5,0.2], approx), \\\n                    error(approx) \\\n                    FROM percentile_agg\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<Vec<f64>, f64>()\n                .unwrap();\n            assert!(\n                test_value\n                    .as_ref()\n                    .unwrap()\n                    .iter()\n                    .zip(value.unwrap())\n                    .all(|(a, b)| { (a - b).abs() < 0.0001 }),\n                \"Some Float value differs from expected by more than {}\",\n                0.0001\n            );\n\n            apx_eql(test_error.unwrap(), error.unwrap(), 0.000001);\n            assert!(test_value\n                .unwrap()\n                .iter()\n                .zip(vec![9.0, 5.0, 2.0])\n                .all(|(a, b)| { matches!(pct_eql(*a, b, test_error.unwrap()), ()) }));\n        });\n    }\n\n    #[pg_test]\n    fn test_approx_percentile_array_arrow() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\n                    \"CREATE TABLE paa_test (device INTEGER, value DOUBLE PRECISION)\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n            client.update(\"INSERT INTO paa_test SELECT dev, dev - v FROM generate_series(1,10) dev, generate_series(0, 1.0, 0.01) v\", None, &[]).unwrap();\n\n            let sanity = client\n                .update(\"SELECT COUNT(*) FROM paa_test\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap();\n            assert_eq!(Some(1010), sanity);\n\n            client\n                .update(\n                    \"CREATE VIEW uddsketch_test AS \\\n                SELECT uddsketch(200, 0.001, value) as approx \\\n                FROM paa_test \",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            client\n                .update(\n                    \"CREATE VIEW percentile_agg AS \\\n                SELECT percentile_agg(value) as approx \\\n                FROM paa_test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            let (value, error) = client\n                .update(\n                    \"SELECT \\\n                    approx_percentile_array(array[0.9,0.5,0.2], approx), \\\n                    error(approx) \\\n                    FROM uddsketch_test\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<Vec<f64>, f64>()\n                .unwrap();\n\n            let (test_value_arrow, test_error_arrow) = client\n                .update(\n                    \"SELECT approx->approx_percentiles(array[0.9,0.5,0.2]), \\\n        \t     error(approx) \\\n                    FROM uddsketch_test\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_two::<Vec<f64>, f64>()\n                .unwrap();\n\n            assert!(\n                test_value_arrow\n                    .as_ref()\n                    .unwrap()\n                    .iter()\n                    .zip(value.as_ref().unwrap())\n                    .all(|(a, b)| { (a - b).abs() < 0.0001 }),\n                \"Some Float value differs from expected by more than {}\",\n                0.0001\n            );\n\n            apx_eql(test_error_arrow.unwrap(), error.unwrap(), 0.000001);\n            assert!(test_value_arrow\n                .unwrap()\n                .iter()\n                .zip(vec![9.0, 5.0, 2.0])\n                .all(|(a, b)| { matches!(pct_eql(*a, b, test_error_arrow.unwrap()), ()) }));\n        });\n    }\n\n    #[pg_test]\n    fn uddsketch_io_test() {\n        Spi::connect_mut(|client| {\n            client\n                .update(\"CREATE TABLE io_test (value DOUBLE PRECISION)\", None, &[])\n                .unwrap();\n            client.update(\"INSERT INTO io_test VALUES (-1000), (-100), (-10), (-1), (-0.1), (-0.01), (-0.001), (0), (0.001), (0.01), (0.1), (1), (10), (100), (1000)\", None, &[]).unwrap();\n\n            let sketch = client\n                .update(\n                    \"SELECT uddsketch(10, 0.01, value)::text FROM io_test\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n\n            let expected = \"(\\\n                version:1,\\\n                alpha:0.9881209712069546,\\\n                max_buckets:10,\\\n                num_buckets:9,\\\n                compactions:8,\\\n                count:15,\\\n                sum:0,\\\n                buckets:[\\\n                    (Negative(2),1),\\\n                    (Negative(1),2),\\\n                    (Negative(0),3),\\\n                    (Negative(-1),1),\\\n                    (Zero,1),\\\n                    (Positive(-1),1),\\\n                    (Positive(0),3),\\\n                    (Positive(1),2),\\\n                    (Positive(2),1)\\\n                    ]\\\n                )\";\n\n            assert_eq!(sketch, Some(expected.into()));\n\n            client\n                .update(\n                    \"CREATE VIEW sketch AS SELECT uddsketch(10, 0.01, value) FROM io_test\",\n                    None,\n                    &[],\n                )\n                .unwrap();\n\n            for cmd in [\n                \"mean(\",\n                \"num_vals(\",\n                \"error(\",\n                \"approx_percentile(0.1,\",\n                \"approx_percentile(0.25,\",\n                \"approx_percentile(0.5,\",\n                \"approx_percentile(0.6,\",\n                \"approx_percentile(0.8,\",\n            ] {\n                let sql1 = format!(\"SELECT {cmd}uddsketch) FROM sketch\");\n                let sql2 = format!(\"SELECT {cmd}'{expected}'::uddsketch) FROM sketch\");\n\n                let expected = client\n                    .update(&sql1, None, &[])\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>()\n                    .unwrap()\n                    .unwrap();\n                let test = client\n                    .update(&sql2, None, &[])\n                    .unwrap()\n                    .first()\n                    .get_one::<f64>()\n                    .unwrap()\n                    .unwrap();\n\n                assert!((expected - test).abs() < f64::EPSILON);\n            }\n        });\n    }\n\n    #[pg_test]\n    fn uddsketch_byte_io_test() {\n        unsafe {\n            use std::ptr;\n            let state = uddsketch_trans_inner(None, 100, 0.005, Some(14.0), ptr::null_mut());\n            let state = uddsketch_trans_inner(state, 100, 0.005, Some(18.0), ptr::null_mut());\n            let state = uddsketch_trans_inner(state, 100, 0.005, Some(22.7), ptr::null_mut());\n            let state = uddsketch_trans_inner(state, 100, 0.005, Some(39.42), ptr::null_mut());\n            let state = uddsketch_trans_inner(state, 100, 0.005, Some(-43.0), ptr::null_mut());\n\n            let control = state.unwrap();\n            let buffer = uddsketch_serialize(Inner::from(control.clone()).internal().unwrap());\n            let buffer = pgrx::varlena::varlena_to_byte_slice(buffer.0.cast_mut_ptr());\n\n            let expected = [\n                1, 1, 123, 20, 174, 71, 225, 122, 116, 63, 100, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5,\n                0, 0, 0, 0, 0, 0, 0, 144, 194, 245, 40, 92, 143, 73, 64, 2, 0, 0, 0, 0, 0, 0, 0,\n                202, 11, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,\n                66, 8, 105, 93, 221, 4, 0, 0, 0, 0, 0, 0, 0, 5, 1, 1, 1,\n            ];\n            assert_eq!(buffer, expected);\n\n            let expected = pgrx::varlena::rust_byte_slice_to_bytea(&expected);\n            let new_state =\n                uddsketch_deserialize_inner(bytea(pg_sys::Datum::from(expected.as_ptr())));\n            assert_eq!(&*new_state, &*control);\n        }\n    }\n\n    #[pg_test]\n    fn test_udd_null_input_yields_null_output() {\n        Spi::connect_mut(|client| {\n            let output = client\n                .update(\"SELECT uddsketch(20, 0.01, NULL)::TEXT\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<String>()\n                .unwrap();\n            assert_eq!(output, None)\n        })\n    }\n}\n"
  },
  {
    "path": "extension/src/utilities.rs",
    "content": "use crate::raw::TimestampTz;\nuse pgrx::prelude::*;\n\n#[pg_extern(\n    name = \"generate_periodic_normal_series\",\n    schema = \"toolkit_experimental\"\n)]\npub fn default_generate_periodic_normal_series(\n    series_start: crate::raw::TimestampTz,\n    rng_seed: Option<i64>,\n) -> TableIterator<'static, (name!(time, TimestampTz), name!(value, f64))> {\n    generate_periodic_normal_series(series_start, None, None, None, None, None, None, rng_seed)\n}\n\n#[allow(clippy::too_many_arguments)]\npub fn alternate_generate_periodic_normal_series(\n    series_start: crate::raw::TimestampTz,\n    periods_per_series: i64,\n    points_per_period: i64,\n    seconds_between_points: i64,\n    base_value: f64,\n    periodic_magnitude: f64,\n    standard_deviation: f64,\n    rng_seed: Option<i64>,\n) -> TableIterator<'static, (name!(time, TimestampTz), name!(value, f64))> {\n    generate_periodic_normal_series(\n        series_start,\n        Some(periods_per_series * points_per_period * seconds_between_points * 1000000),\n        Some(seconds_between_points * 1000000),\n        Some(base_value),\n        Some(points_per_period * seconds_between_points * 1000000),\n        Some(periodic_magnitude),\n        Some(standard_deviation),\n        rng_seed,\n    )\n}\n\n#[allow(clippy::too_many_arguments)]\n#[pg_extern(schema = \"toolkit_experimental\")]\npub fn generate_periodic_normal_series(\n    series_start: crate::raw::TimestampTz,\n    series_len: Option<i64>,      //pg_sys::Interval,\n    sample_interval: Option<i64>, //pg_sys::Interval,\n    base_value: Option<f64>,\n    period: Option<i64>, //pg_sys::Interval,\n    periodic_magnitude: Option<f64>,\n    standard_deviation: Option<f64>,\n    rng_seed: Option<i64>,\n) -> TableIterator<'static, (name!(time, TimestampTz), name!(value, f64))> {\n    // Convenience consts to make defaults more readable\n    const SECOND: i64 = 1000000;\n    const MIN: i64 = 60 * SECOND;\n    const HOUR: i64 = 60 * MIN;\n    const DAY: i64 = 24 * HOUR;\n\n    // TODO: exposing defaults in the PG function definition would be much nicer\n    let series_len = series_len.unwrap_or(28 * DAY);\n    let sample_interval = sample_interval.unwrap_or(10 * MIN);\n    let base_value = base_value.unwrap_or(1000.0);\n    let period = period.unwrap_or(DAY);\n    let periodic_magnitude = periodic_magnitude.unwrap_or(100.0);\n    let standard_deviation = standard_deviation.unwrap_or(100.0);\n\n    use rand::SeedableRng;\n    use rand_chacha::ChaCha12Rng;\n    use rand_distr::Distribution;\n\n    let mut rng = match rng_seed {\n        Some(v) => ChaCha12Rng::seed_from_u64(v as u64),\n        None => ChaCha12Rng::from_entropy(),\n    };\n\n    let distribution = rand_distr::Normal::new(0.0, standard_deviation).unwrap();\n\n    let series_start: i64 = series_start.into();\n    TableIterator::new(\n        (0..series_len)\n            .step_by(sample_interval as usize)\n            .map(move |accum| {\n                let time = series_start + accum;\n                let base = base_value\n                    + f64::sin(accum as f64 / (2.0 * std::f64::consts::PI * period as f64))\n                        * periodic_magnitude;\n                let error = distribution.sample(&mut rng);\n                (time.into(), base + error)\n            }),\n    )\n}\n\n// Returns days in month\nextension_sql!(\n    \"\nCREATE FUNCTION days_in_month(date timestamptz) RETURNS int\nSET search_path TO pg_catalog,pg_temp\nAS $$\nSELECT CAST(EXTRACT('day' FROM (date_trunc('month', $1) + interval '1 month' - date_trunc('month', $1))) AS INTEGER)\n$$ LANGUAGE SQL STRICT IMMUTABLE PARALLEL SAFE;\n\",\n    name = \"days_in_month\",\n);\n\n// Normalizes metric based on reference date and days\nextension_sql!(\n    \"\nCREATE FUNCTION month_normalize(metric float8, reference_date timestamptz, days float8 DEFAULT 365.25/12) RETURNS float8\nSET search_path TO pg_catalog,pg_temp\nAS $$\nSELECT metric * days / CAST(EXTRACT('day' FROM (reference_date + interval '1 month' - reference_date)) as INTEGER)\n$$ LANGUAGE SQL STRICT IMMUTABLE PARALLEL SAFE;\n\",\n    name=\"month_normalize\",\n);\n\n// Convert a timestamp to a double precision unix epoch\nextension_sql!(\n    \"\nCREATE FUNCTION to_epoch(timestamptz) RETURNS DOUBLE PRECISION LANGUAGE SQL IMMUTABLE PARALLEL SAFE\nSET search_path TO pg_catalog,pg_temp\nAS $$\nSELECT EXTRACT(EPOCH FROM $1);\n$$;\n\",\n    name = \"to_epoch\",\n);\n\n#[cfg(any(test, feature = \"pg_test\"))]\n#[pg_schema]\nmod tests {\n    use pgrx::*;\n    use pgrx_macros::pg_test;\n\n    #[pg_test]\n    fn test_to_epoch() {\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT to_epoch('2021-01-01 00:00:00+03'::timestamptz)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap();\n            assert!((test_val - 1609448400f64).abs() < f64::EPSILON);\n\n            let test_val = client\n                .update(\"SELECT to_epoch('epoch'::timestamptz)\", None, &[])\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap();\n            assert!((test_val - 0f64).abs() < f64::EPSILON);\n\n            let test_val = client\n                .update(\n                    \"SELECT to_epoch('epoch'::timestamptz - interval '42 seconds')\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap();\n            assert!((test_val - -42f64).abs() < f64::EPSILON);\n        });\n    }\n\n    #[pg_test]\n    fn test_days_in_month() {\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT days_in_month('2021-01-01 00:00:00+03'::timestamptz)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test_val, 31);\n        });\n\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT days_in_month('2020-02-03 00:00:00+03'::timestamptz)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test_val, 29);\n        });\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT days_in_month('2023-01-31 00:00:00+00'::timestamptz)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<i64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test_val, 31);\n        });\n    }\n    #[pg_test]\n    fn test_monthly_normalize() {\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT month_normalize(1000,'2021-01-01 00:00:00+03'::timestamptz)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test_val, 981.8548387096774f64);\n        });\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT month_normalize(1000,'2021-01-01 00:00:00+03'::timestamptz,30.5)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test_val, 983.8709677419355f64);\n        });\n        Spi::connect_mut(|client| {\n            let test_val = client\n                .update(\n                    \"SELECT month_normalize(1000,'2021-01-01 00:00:00+03'::timestamptz,30)\",\n                    None,\n                    &[],\n                )\n                .unwrap()\n                .first()\n                .get_one::<f64>()\n                .unwrap()\n                .unwrap();\n            assert_eq!(test_val, 967.741935483871f64);\n        });\n    }\n}\n"
  },
  {
    "path": "extension/timescaledb_toolkit.control",
    "content": "comment = 'Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities'\ndefault_version = '@CARGO_VERSION@'\nrelocatable = false\nsuperuser = false\nmodule_pathname = '$libdir/timescaledb_toolkit' # only for testing, will be removed for real installs\n# comma-separated list of previous versions this version can be upgraded from\n# directly. This is used to generate upgrade scripts.\n# upgradeable_from = '1.6.0, 1.7.0, 1.8.0, 1.10.0-dev, 1.10.1, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.21.0'\n"
  },
  {
    "path": "tests/update/candlestick.md",
    "content": "# Candlestick Tests\n\n## Get candlestick values from tick data\n\n\n```sql,creation,min-toolkit-version=1.14.0\nCREATE TABLE stocks_real_time(time TIMESTAMPTZ, symbol TEXT, price DOUBLE PRECISION,day_volume DOUBLE PRECISION);\nINSERT INTO stocks_real_time VALUES\n    ('2023-01-11',\t'AAPL',\t133.445,10),\t\n    ('2023-01-11',\t'PFE',    47.38,2),\t\n    ('2023-01-11',\t'AMZN',\t95.225,1),\n    ('2023-01-11',\t'INTC',\t29.82,NULL),\n    ('2023-01-11',\t'MSFT',\t235.5,100),\n    ('2023-01-11',\t'TSLA',\t123.085,NULL),\t\n    ('2023-01-11',\t'AAPL',\t133.44,20);\n\nCREATE MATERIALIZED VIEW cs AS\n    SELECT symbol,\n        candlestick_agg(\"time\", price, day_volume) AS candlestick\n    FROM stocks_real_time\n    GROUP BY symbol;\n```\n\n```sql,validation,min-toolkit-version=1.14.0\nSELECT\n  symbol,\n  open(candlestick),\n  high(candlestick),\n  low(candlestick),\n  close(candlestick),\n  volume(candlestick)\nFROM cs\nORDER BY symbol;\n```\n\n```output\n symbol |  open   |  high   |   low   |  close  | volume\n--------+---------+---------+---------+---------+--------\n AAPL   | 133.445 | 133.445 |  133.44 | 133.445 |     30\n AMZN   |  95.225 |  95.225 |  95.225 |  95.225 |      1\n INTC   |   29.82 |   29.82 |   29.82 |   29.82 |\n MSFT   |   235.5 |   235.5 |   235.5 |   235.5 |    100\n PFE    |   47.38 |   47.38 |   47.38 |   47.38 |      2\n TSLA   | 123.085 | 123.085 | 123.085 | 123.085 |\n ```\n"
  },
  {
    "path": "tests/update/heartbeat.md",
    "content": "# Candlestick Tests\n\n## Get candlestick values from tick data\n\n\n```sql,creation,min-toolkit-version=1.15.0\nCREATE TABLE liveness(heartbeat TIMESTAMPTZ, start TIMESTAMPTZ);\nINSERT INTO liveness VALUES\n    ('01-01-2020 0:2:20 UTC', '01-01-2020 0:0 UTC'),\n    ('01-01-2020 0:10 UTC', '01-01-2020 0:0 UTC'),\n    ('01-01-2020 0:17 UTC', '01-01-2020 0:0 UTC'),\n    ('01-01-2020 0:30 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 0:35 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 0:40 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 0:35 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 0:40 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 0:40 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 0:50:30 UTC', '01-01-2020 0:30 UTC'),\n    ('01-01-2020 1:00:30 UTC', '01-01-2020 1:00 UTC'),\n    ('01-01-2020 1:08 UTC', '01-01-2020 1:00 UTC'),\n    ('01-01-2020 1:18 UTC', '01-01-2020 1:00 UTC'),\n    ('01-01-2020 1:28 UTC', '01-01-2020 1:00 UTC'),\n    ('01-01-2020 1:38:01 UTC', '01-01-2020 1:30 UTC'),\n    ('01-01-2020 1:40 UTC', '01-01-2020 1:30 UTC'),\n    ('01-01-2020 1:40:01 UTC', '01-01-2020 1:30 UTC'),\n    ('01-01-2020 1:50:01 UTC', '01-01-2020 1:30 UTC'),\n    ('01-01-2020 1:57 UTC', '01-01-2020 1:30 UTC'),\n    ('01-01-2020 1:59:50 UTC', '01-01-2020 1:30 UTC');\n\nCREATE MATERIALIZED VIEW hb AS\n    SELECT start,\n        heartbeat_agg(heartbeat, start, '30m', '10m') AS agg\n    FROM liveness\n    GROUP BY start;\n```\n\n```sql,validation,min-toolkit-version=1.15.0\nSELECT\n  start,\n  uptime(agg),\n  interpolated_uptime(agg, LAG(agg) OVER (ORDER by start))\nFROM hb\nORDER BY start;\n```\n\n```output\n         start          |  uptime  | interpolated_uptime \n------------------------+----------+---------------------\n 2020-01-01 00:00:00+00 | 00:24:40 | 00:24:40\n 2020-01-01 00:30:00+00 | 00:29:30 | 00:29:30\n 2020-01-01 01:00:00+00 | 00:29:30 | 00:30:00\n 2020-01-01 01:30:00+00 | 00:21:59 | 00:29:59\n ```\n"
  },
  {
    "path": "tests/update/original_update_tests.md",
    "content": "# Original Update Tests\n\n\n\n```sql,creation,min-toolkit-version=1.4.0\nCREATE TABLE test_data(ts timestamptz, val DOUBLE PRECISION);\n    INSERT INTO test_data\n        SELECT '2020-01-01 00:00:00+00'::timestamptz + i * '1 hour'::interval,\n        100 + i % 100\n    FROM generate_series(0, 10000) i;\n\t\nCREATE MATERIALIZED VIEW regression_view AS\n    SELECT\n        counter_agg(ts, val) AS countagg,\n        hyperloglog(1024, val) AS hll,\n        time_weight('locf', ts, val) AS twa,\n        uddsketch(100, 0.001, val) as udd,\n        tdigest(100, val) as tdig,\n        stats_agg(val) as stats\n    FROM test_data;\n```\n\n\n\n```sql,validation,min-toolkit-version=1.4.0\nSELECT\n    num_resets(countagg),\n    distinct_count(hll),\n    average(twa),\n    approx_percentile(0.1, udd),\n    approx_percentile(0.1, tdig),\n    kurtosis(stats)\nFROM regression_view;\n```\n\n```output\n num_resets | distinct_count | average | approx_percentile  | approx_percentile  |      kurtosis\n------------+----------------+---------+--------------------+--------------------+--------------------\n        100 |            100 |   149.5 | 108.96220333142547 | 109.50489521100047 | 1.7995661075080858\n```\n"
  },
  {
    "path": "tests/update/state_agg.md",
    "content": "# `state_agg` tests\n\n```sql,creation,min-toolkit-version=1.15.0\nCREATE TABLE states_test(ts TIMESTAMPTZ, state TEXT);\nINSERT INTO states_test VALUES\n    ('2020-01-01 00:00:00+00', 'START'),\n    ('2020-01-01 00:00:11+00', 'OK'),\n    ('2020-01-01 00:01:00+00', 'ERROR'),\n    ('2020-01-01 00:01:03+00', 'OK'),\n    ('2020-01-01 00:02:00+00', 'STOP');\n\nCREATE TABLE agg(sa StateAgg);\nINSERT INTO agg SELECT state_agg(ts, state) FROM states_test;\n```\n\n```sql,validation,min-toolkit-version=1.15.0\nSELECT (state_timeline(sa)).* FROM agg;\n```\n```output\n state |       start_time       |        end_time\n-------+------------------------+------------------------\n START | 2020-01-01 00:00:00+00 | 2020-01-01 00:00:11+00\n OK    | 2020-01-01 00:00:11+00 | 2020-01-01 00:01:00+00\n ERROR | 2020-01-01 00:01:00+00 | 2020-01-01 00:01:03+00\n OK    | 2020-01-01 00:01:03+00 | 2020-01-01 00:02:00+00\n STOP  | 2020-01-01 00:02:00+00 | 2020-01-01 00:02:00+00\n```\n"
  },
  {
    "path": "tests/update/time-vector.md",
    "content": "# Time Vector Tests\n\n```sql,creation\nCREATE TABLE time_vector_data(time TIMESTAMPTZ, value DOUBLE PRECISION);\nINSERT INTO time_vector_data VALUES\n    ('2020-1-1 UTC', 30.0),\n    ('2020-1-2 UTC', 45.0),\n    ('2020-1-3 UTC', NULL),\n    ('2020-1-4 UTC', 55.5),\n    ('2020-1-5 UTC', 10.0);\n```\n\n```sql,validation\nSELECT unnest(timevector(time,value))::TEXT FROM time_vector_data;\n```\n\n```output\n             unnest\n---------------------------------\n (\"2020-01-01 00:00:00+00\",30)\n (\"2020-01-02 00:00:00+00\",45)\n (\"2020-01-03 00:00:00+00\",NaN)\n (\"2020-01-04 00:00:00+00\",55.5)\n (\"2020-01-05 00:00:00+00\",10)\n ```\n\n```sql,creation\nCREATE TABLE tv_rollup_data(time TIMESTAMPTZ, value DOUBLE PRECISION, bucket INTEGER);\nINSERT INTO tv_rollup_data VALUES\n    ('2020-1-1 UTC', 30.0, 1),\n    ('2020-1-2 UTC', 45.0, 1),\n    ('2020-1-3 UTC', NULL, 2),\n    ('2020-1-4 UTC', 55.5, 2),\n    ('2020-1-5 UTC', 10.0, 3),\n    ('2020-1-6 UTC', 13.0, 3),\n    ('2020-1-7 UTC', 71.0, 4),\n    ('2020-1-8 UTC', 0.0, 4);\n```\n\n```sql,validation\nSELECT unnest(rollup(tvec))::TEXT\n   FROM (\n       SELECT timevector(time, value) AS tvec\n       FROM tv_rollup_data \n       GROUP BY bucket \n       ORDER BY bucket\n   ) s;\n```\n\n```output\n            unnest\n-------------------------------\n (\"2020-01-01 00:00:00+00\",30)\n (\"2020-01-02 00:00:00+00\",45)\n (\"2020-01-03 00:00:00+00\",NaN)\n (\"2020-01-04 00:00:00+00\",55.5)\n (\"2020-01-05 00:00:00+00\",10)\n (\"2020-01-06 00:00:00+00\",13)\n (\"2020-01-07 00:00:00+00\",71)\n (\"2020-01-08 00:00:00+00\",0)\n ```\n"
  },
  {
    "path": "tests/update/time-weighted-average.md",
    "content": "# Time Weighted Average Tests\n\n## Test integral and interpolated integral\n\n```sql,creation,min-toolkit-version=1.15.0\nCREATE TABLE time_weight_test(time timestamptz, value double precision, bucket timestamptz);\nINSERT INTO time_weight_test VALUES\n    ('2020-1-1 8:00'::timestamptz, 10.0, '2020-1-1'::timestamptz),\n    ('2020-1-1 12:00'::timestamptz, 40.0, '2020-1-1'::timestamptz),\n    ('2020-1-1 16:00'::timestamptz, 20.0, '2020-1-1'::timestamptz),\n    ('2020-1-2 2:00'::timestamptz, 15.0, '2020-1-2'::timestamptz),\n    ('2020-1-2 12:00'::timestamptz, 50.0, '2020-1-2'::timestamptz),\n    ('2020-1-2 20:00'::timestamptz, 25.0, '2020-1-2'::timestamptz),\n    ('2020-1-3 10:00'::timestamptz, 30.0, '2020-1-3'::timestamptz),\n    ('2020-1-3 12:00'::timestamptz, 0.0, '2020-1-3'::timestamptz), \n    ('2020-1-3 16:00'::timestamptz, 35.0, '2020-1-3'::timestamptz);\nCREATE MATERIALIZED VIEW twa AS (\n    SELECT bucket, time_weight('linear', time, value) as agg \n    FROM time_weight_test \n    GROUP BY bucket\n);\n```\n\n```sql,validation,min-toolkit-version=1.15.0\nSELECT\n    bucket,\n    interpolated_integral(\n        agg,\n        bucket,\n        '1 day'::interval, \n        LAG(agg) OVER (ORDER BY bucket),\n        LEAD(agg) OVER (ORDER BY bucket),\n        'hours')\nFROM twa\nORDER BY bucket;\n```\n\n```output\n         bucket         | interpolated_integral\n------------------------+-----------------------\n 2020-01-01 00:00:00+00 |                   364\n 2020-01-02 00:00:00+00 |     758.8571428571429\n 2020-01-03 00:00:00+00 |     382.1428571428571\n```\n\n```sql,validation,min-toolkit-version=1.15.0\nSELECT bucket, integral(agg, 'hrs') FROM twa ORDER BY bucket;\n```\n\n```output\n         bucket         | integral\n------------------------+----------\n 2020-01-01 00:00:00+00 |      220\n 2020-01-02 00:00:00+00 |      625\n 2020-01-03 00:00:00+00 |      100\n```\n"
  },
  {
    "path": "tools/build",
    "content": "#!/bin/sh\n\nset -ex\n\nprint() {\n    printf '%s\\n' \"$*\"\n}\n\ndie() {\n    st=${?:-0}\n    if [ $st -eq 0 ]; then\n        st=2\n    fi\n    print \"$*\" >&2\n    exit $st\n}\n\nusage() {\n    die 'build [ -n -pg1[234] -profile release ] ( test-crates | test-extension | install | test-doc | test-updates | clippy)'\n}\n\nrequire_pg_version() {\n    [ -n \"$pg_version\" ] || die 'specify one of -pg15 | -pg16 | -pg17 | -pg18'\n}\n\nfind_pg_config() {\n    if [ -z \"$pg_config\" ]; then\n        require_pg_version\n        pg_config=`which $(sed -ne 's/\"//g' -e \"s/^pg$pg_version *= *//p\" ~/.pgrx/config.toml)`\n    fi\n    [ -x \"$pg_config\" ] || die \"$pg_config not executable\"\n}\n\nrequire_cargo_pgrx() {\n    [ -n \"$cargo_pgrx\" ] || die 'specify path to cargo-pgrx (0.4 series or newer)'\n}\n\nrequire_cargo_pgrx_old() {\n    [ -n \"$cargo_pgrx_old\" ] || die 'specify path to cargo-pgrx (0.2-0.3 series)'\n}\n\nfind_profile() {\n    [ -n \"$profile\" ] || profile=dev\n}\n\n[ $# -ge 1 ] || usage\n\n# check versions\ncargo --version\nrustc --version\nrustup --version\n\nwhile [ $# -gt 0 ]; do\n    arg=\"$1\"\n    shift\n    case \"$arg\" in\n        -n)\n            nop=:\n            ;;\n\n        -pgconfig)\n            pg_config=\"$1\"\n            shift\n            ;;\n\n        -cargo-pgrx)\n            cargo_pgrx=\"$1\"\n            shift\n            ;;\n\n        -cargo-pgrx-old)\n            cargo_pgrx_old=\"$1\"\n            shift\n            ;;\n\n        -pgport)\n            pg_port=\"$1\"\n            shift\n            ;;\n\n        -pg1[0-9])         # If this script survives to postgresql 19, WE WIN!\n            pg_version=${arg#-pg}\n            pg=pg$pg_version\n            [ -z \"$pg_port\" ] && pg_port=288$pg_version\n            ;;\n\n        -profile)\n            profile=\"$1\"\n            shift\n            ;;\n\n        clippy)\n            find_profile\n            $nop cargo fetch\n            $nop cargo clippy --profile $profile --workspace --features pg_test -- -D warnings\n            ;;\n\n        test-crates)\n            # Should find no dependency crates to fetch.  If it finds any, we need to update the cache key.\n            find_profile\n            $nop cargo fetch\n            $nop cargo test --profile $profile --workspace --exclude timescaledb_toolkit\n            ;;\n\n        test-extension)\n            cd extension\n            find_profile\n            require_pg_version\n            $nop cargo fetch\n            $nop cargo test --profile $profile --features \"$pg pg_test\" --no-default-features\n            ;;\n\n        install)\n            find_profile\n            require_pg_version\n            find_pg_config\n            (cd extension && $nop cargo pgrx install --profile $profile -c \"$pg_config\")\n            $nop cargo run --manifest-path tools/post-install/Cargo.toml \"$pg_config\"\n            ;;\n\n        test-doc)\n            find_profile\n            require_pg_version\n            $nop cargo pgrx start --package timescaledb_toolkit $pg || (cat /home/postgres/.pgrx/${pg_version}.log; false)\n            $nop cargo run --profile $profile -p sql-doctester -- \\\n                 -h localhost \\\n                 -p $pg_port \\\n                 docs\n            $nop cargo pgrx stop --package timescaledb_toolkit $pg\n            ;;\n\n        test-updates)\n            find_profile\n            require_pg_version\n            find_pg_config\n            require_cargo_pgrx\n            require_cargo_pgrx_old\n            $nop cargo pgrx start --package timescaledb_toolkit $pg || (cat /home/postgres/.pgrx/${pg_version}.log; false)\n            $nop cargo run --profile $profile --manifest-path tools/update-tester/Cargo.toml -- full-update-test-source \\\n                 -h localhost \\\n                 -p $pg_port \\\n                 --cache old-versions \\\n                 \"$pg_config\" \\\n                 \"$cargo_pgrx\" \\\n                 \"$cargo_pgrx_old\"\n            $nop cargo pgrx stop --package timescaledb_toolkit $pg\n            ;;\n\n        *)\n            usage\n            ;;\n    esac\ndone\n"
  },
  {
    "path": "tools/dependencies.sh",
    "content": "# Dependency configuration\n# Ideally, all dependencies would be specified in just one place.\n# Exceptions:\n# - crate dependencies are specified in Cargo.toml files\n# - postgres versions are duplicated in the Github Actions matrix\n# - Readme.md lists some, too.  TODO is it acceptable to just point to this file?\n# All our automation scripts read this, so at least we're not duplicating this\n# information across all those.\n\nPG_VERSIONS='15 16 17 18'\n# TODO: extend this with 18 this once TimescaleDB supports PostgreSQL 18\nTSDB_PG_VERSIONS='15 16 17'\n\nCARGO_EDIT=0.11.2\n\n# Keep synchronized with extension/Cargo.toml and `cargo install --version N.N.N cargo-pgrx` in Readme.md .\nPGRX_VERSION=0.16.1\n\nRUST_TOOLCHAIN=1.89.0\nRUST_PROFILE=minimal\nRUST_COMPONENTS=clippy,rustfmt\n\n# We use fpm 1.14.2 to build RPMs.\n# TODO Use rpmbuild directly.\nFPM_VERSION=1.14.2\n\nGH_DEB_URL=https://github.com/cli/cli/releases/download/v2.16.1/gh_2.16.1_linux_amd64.deb\nGH_DEB_SHA256=d0ba8693b6e4c1bde6683ccfa971a15c00b9fe92865074d48609959d04399dc7\n"
  },
  {
    "path": "tools/install-timescaledb",
    "content": "#!/bin/sh\ngit clone \"$2\" timescaledb\ncd timescaledb\ngit switch --detach \"$3\"\nmkdir build\ncd build\n# this overwrites the files from the TimescaleDB package\ncmake .. -DUSE_OPENSSL=0 -DLINTER=0 -DCMAKE_PROGRAM_PATH=/usr/lib/postgresql/$1/bin\nmake install\n"
  },
  {
    "path": "tools/post-install/Cargo.toml",
    "content": "[package]\nname = \"post-install\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nxshell = \"0.1.17\"\nwalkdir = \"2\""
  },
  {
    "path": "tools/post-install/src/main.rs",
    "content": "#![allow(unexpected_cfgs)]\n\nuse std::{\n    env,\n    fs::{self, File},\n    io::{BufRead, BufReader, BufWriter, Write},\n    path::{Path, PathBuf},\n    process,\n};\n\nuse xshell::cmd;\n\nmod update_script;\n\nmacro_rules! path {\n    ($start:ident $(/ $segment: literal)*) => {\n        {\n            let root: &Path = $start.as_ref();\n            root $(.join($segment))*\n        }\n    };\n    ($start:ident / $segment: expr) => {\n        {\n            let root: &Path = $start.as_ref();\n            root.join($segment)\n        }\n    }\n}\n\nfn main() {\n    if let Err(err) = try_main() {\n        eprintln!(\"{err}\");\n        process::exit(1);\n    }\n}\n\nfn try_main() -> xshell::Result<()> {\n    let pg_config = env::args().nth(1).expect(\"missing /path/to/pg_config\");\n    let extension_info = if pg_config == \"--dir\" {\n        let package_dir = env::args().nth(2).expect(\"missing /path/to/package_dir\");\n        get_extension_info_from_dir(&package_dir)?\n    } else {\n        get_extension_info_from_pg_config(&pg_config)?\n    };\n\n    // remove `module_path = '$libdir/timescaledb_toolkit'`\n    // from timescaledb_toolkit.control.\n    // Not needed for correctness purposes, but it ensures that if `MODULE_PATH`\n    // is left anywhere in the install script, it will fail to install.\n    remove_module_path_from_control_file(&extension_info);\n\n    // rename timescaledb_toolkit.so to timescaledb_toolkit-<current version>.so\n    add_version_to_binary(&extension_info);\n\n    // replace `MODULE_PATH` with `$libdir/timescaledb_toolkit-<current version>`\n    add_version_to_install_script(&extension_info);\n\n    generate_update_scripts(&extension_info);\n\n    Ok(())\n}\n\nstruct ExtensionInfo {\n    control_file: PathBuf,\n    current_version: String,\n    upgradeable_from: Vec<String>,\n    bin_dir: PathBuf,\n    extension_dir: PathBuf,\n}\n\nfn get_extension_info_from_pg_config(pg_config: &str) -> xshell::Result<ExtensionInfo> {\n    let bin_dir = cmd!(\"{pg_config} --pkglibdir\").read()?;\n\n    let share_dir = cmd!(\"{pg_config} --sharedir\").read()?;\n    let extension_dir = path!(share_dir / \"extension\");\n\n    let control_file = path!(extension_dir / \"timescaledb_toolkit.control\");\n\n    let control_contents = fs::read_to_string(&control_file).unwrap_or_else(|e| {\n        panic!(\n            \"cannot read control file {} due to {e}\",\n            control_file.to_string_lossy()\n        )\n    });\n\n    let current_version = get_current_version(&control_contents);\n    eprintln!(\"Generating Version {current_version}\");\n\n    let upgradeable_from = get_upgradeable_from(&control_contents);\n    eprintln!(\"Upgradable From {upgradeable_from:?}\");\n\n    let extension_info = ExtensionInfo {\n        control_file,\n        current_version,\n        upgradeable_from,\n        bin_dir: bin_dir.into(),\n        extension_dir,\n    };\n    Ok(extension_info)\n}\n\nfn get_extension_info_from_dir(root: &str) -> xshell::Result<ExtensionInfo> {\n    use std::ffi::OsStr;\n\n    let walker = walkdir::WalkDir::new(root).contents_first(true);\n\n    let mut extension_info = None;\n    let mut bin_dir = None;\n    for entry in walker {\n        let entry = entry.unwrap();\n        if entry.file_type().is_file() {\n            let path = entry.into_path();\n            if path.extension() == Some(OsStr::new(\"control\")) {\n                // found the control file\n                let extension_dir = path\n                    .parent()\n                    .expect(\"control file not in dir\")\n                    .to_path_buf();\n                extension_info = Some((extension_dir, path));\n            } else if path.extension() == Some(OsStr::new(\"so\")) {\n                // found the binary\n                bin_dir = Some(path.parent().expect(\"binary file not in dir\").to_path_buf());\n            }\n            if extension_info.is_some() && bin_dir.is_some() {\n                break;\n            }\n        }\n    }\n    if bin_dir.is_none() || extension_info.is_none() {\n        panic!(\"could not find extension objects\")\n    }\n\n    let bin_dir = bin_dir.unwrap();\n\n    let (extension_dir, control_file) = extension_info.unwrap();\n\n    let control_contents = fs::read_to_string(&control_file).unwrap_or_else(|e| {\n        panic!(\n            \"cannot read control file {} due to {e}\",\n            control_file.to_string_lossy()\n        )\n    });\n\n    let current_version = get_current_version(&control_contents);\n    eprintln!(\"Generating Version {current_version}\");\n\n    let upgradeable_from = get_upgradeable_from(&control_contents);\n    eprintln!(\"Upgradable From {upgradeable_from:?}\");\n\n    let extension_info = ExtensionInfo {\n        control_file,\n        current_version,\n        upgradeable_from,\n        bin_dir,\n        extension_dir,\n    };\n    Ok(extension_info)\n}\n\nfn get_current_version(control_contents: &str) -> String {\n    get_field_val(control_contents, \"default_version\").to_string()\n}\n\nfn get_upgradeable_from(control_contents: &str) -> Vec<String> {\n    // versions is a comma-delimited list of versions\n    let versions = get_field_val(control_contents, \"upgradeable_from\");\n    versions\n        .split_terminator(',')\n        .map(|version| version.trim().to_string())\n        .collect()\n}\n\nfn remove_module_path_from_control_file(ExtensionInfo { control_file, .. }: &ExtensionInfo) {\n    let tmp_file = control_file.with_extension(\"control.tmp\");\n    transform_file_to(control_file, &tmp_file, |line| {\n        if line.starts_with(\"module_pathname\") {\n            return \"\".to_string();\n        }\n\n        line\n    });\n    rename_file(tmp_file, control_file);\n}\n\nfn add_version_to_binary(\n    ExtensionInfo {\n        current_version,\n        bin_dir,\n        ..\n    }: &ExtensionInfo,\n) {\n    let bin_file = path!(bin_dir / \"timescaledb_toolkit.so\");\n    let versioned_file = path!(bin_dir / format!(\"timescaledb_toolkit-{}.so\", current_version));\n    rename_file(bin_file, versioned_file);\n}\n\nfn add_version_to_install_script(\n    ExtensionInfo {\n        current_version,\n        extension_dir,\n        ..\n    }: &ExtensionInfo,\n) {\n    let install_script =\n        path!(extension_dir / format!(\"timescaledb_toolkit--{current_version}.sql\"));\n\n    let versioned_script = install_script.with_extension(\"sql.tmp\");\n\n    let module_path = format!(\"$libdir/timescaledb_toolkit-{current_version}\");\n\n    transform_file_to(&install_script, &versioned_script, |line| {\n        assert!(\n            !line.contains(\"CREATE OR REPLACE FUNCTION\"),\n            \"pgrx should not generate CREATE OR REPLACE in functions\"\n        );\n        if line.contains(\"MODULE_PATHNAME\") {\n            return line.replace(\"MODULE_PATHNAME\", &module_path);\n        }\n        line\n    });\n\n    rename_file(&versioned_script, &install_script);\n}\n\n//\n// upgrade scripts\n//\n\nfn generate_update_scripts(\n    ExtensionInfo {\n        current_version,\n        upgradeable_from,\n        extension_dir,\n        ..\n    }: &ExtensionInfo,\n) {\n    let extension_path =\n        path!(extension_dir / format!(\"timescaledb_toolkit--{}.sql\", current_version));\n\n    for from_version in upgradeable_from {\n        let mut extension_file = open_file(&extension_path);\n\n        let upgrade_path = path!(\n            extension_dir\n                / format!(\n                    \"timescaledb_toolkit--{from}--{to}.sql\",\n                    from = from_version,\n                    to = current_version\n                )\n        );\n        let mut upgrade_file = create_file(&upgrade_path);\n\n        update_script::generate_from_install(\n            from_version,\n            current_version,\n            &mut extension_file,\n            &mut upgrade_file,\n        );\n\n        copy_permissions(extension_file, upgrade_file);\n    }\n}\n\ntrait PushLine {\n    fn push_line(&mut self, line: &str);\n}\n\nimpl PushLine for String {\n    fn push_line(&mut self, line: &str) {\n        self.push_str(line);\n        self.push('\\n');\n    }\n}\n\n//\n// control file utils\n//\n\n// find a `<field name> = '<field value>'` and extract `<field value>`\nfn get_field_val<'a>(contents: &'a str, field: &str) -> &'a str {\n    contents\n        .lines()\n        .filter(|line| line.contains(field))\n        .map(get_quoted_field)\n        .next()\n        .unwrap_or_else(|| panic!(\"cannot read field `{field}` in control file\"))\n}\n\n// given a `<field name> = '<field value>'` extract `<field value>`\nfn get_quoted_field(line: &str) -> &str {\n    let quoted = line\n        .split('=')\n        .nth(1)\n        .unwrap_or_else(|| panic!(\"cannot find value in line `{line}`\"));\n\n    quoted\n        .trim_start()\n        .split_terminator('\\'')\n        .find(|s| !s.is_empty())\n        .unwrap_or_else(|| panic!(\"unquoted value in line `{line}`\"))\n}\n\n//\n// file utils\n//\n\nfn open_file(path: impl AsRef<Path>) -> BufReader<File> {\n    let path = path.as_ref();\n    let file = File::open(path)\n        .unwrap_or_else(|e| panic!(\"cannot open file `{}` due to {e}\", path.to_string_lossy()));\n    BufReader::new(file)\n}\n\nfn create_file(path: impl AsRef<Path>) -> BufWriter<File> {\n    let path = path.as_ref();\n    let file = File::create(path)\n        .unwrap_or_else(|e| panic!(\"cannot create file `{}` due to {e}\", path.to_string_lossy()));\n    BufWriter::new(file)\n}\n\nfn rename_file(from: impl AsRef<Path>, to: impl AsRef<Path>) {\n    let from = from.as_ref();\n    let to = to.as_ref();\n    fs::rename(from, to).unwrap_or_else(|e| {\n        panic!(\n            \"cannot rename `{}` to `{}` due to `{e}`\",\n            from.to_string_lossy(),\n            to.to_string_lossy()\n        )\n    });\n}\n\nfn transform_file_to(\n    from: impl AsRef<Path>,\n    to: impl AsRef<Path>,\n    mut transform: impl FnMut(String) -> String,\n) {\n    let to_path = to.as_ref();\n    let mut to = create_file(to_path);\n    let from_path = from.as_ref();\n    let mut from = open_file(from_path);\n\n    for line in (&mut from).lines() {\n        let line = line\n            .unwrap_or_else(|e| panic!(\"cannot read `{}` due to {e}\", from_path.to_string_lossy()));\n\n        writeln!(&mut to, \"{}\", transform(line)).unwrap_or_else(|e| {\n            panic!(\"cannot write to `{}` due to {e}\", to_path.to_string_lossy())\n        });\n    }\n\n    copy_permissions(from, to);\n}\n\nfn copy_permissions(from: BufReader<File>, to: BufWriter<File>) {\n    let permissions = from.into_inner().metadata().unwrap().permissions();\n    to.into_inner()\n        .unwrap()\n        .set_permissions(permissions)\n        .unwrap();\n}\n"
  },
  {
    "path": "tools/post-install/src/update_script.rs",
    "content": "use std::{\n    collections::HashSet,\n    io::{BufRead, Write},\n    iter::Peekable,\n};\n\nuse crate::PushLine;\n\nstatic ALTERABLE_PROPERTIES: [&str; 6] = [\n    \"RECEIVE\",\n    \"SEND\",\n    \"TYPMOD_IN\",\n    \"TYPMOD_OUT\",\n    \"ANALYZE\",\n    \"STORAGE\",\n];\n\n#[path = \"../../../extension/src/stabilization_info.rs\"]\nmod stabilization_info;\n\n// our update script is a copy of the install script with the following changes\n// 1. we drop the experimental schema so everything inside it is dropped.\n// 2. drop the event triggers in case we're coming from a version that had them\n// 3. for all CREATEs we check if the object is new in `current_version`\n//     a. if it is, we output the CREATE as-is\n//     b. if it's not, we output the equivalent REPLACE, if one is needed\npub(crate) fn generate_from_install(\n    from_version: &str,\n    current_version: &str,\n    extension_file: impl BufRead,\n    mut upgrade_file: impl Write,\n) {\n    let new_stabilizations = new_stabilizations(from_version, current_version);\n\n    writeln!(\n        &mut upgrade_file,\n        \"DROP SCHEMA IF EXISTS toolkit_experimental CASCADE;\\n\\\n        -- drop the EVENT TRIGGERs; there's no CREATE OR REPLACE for those\n        DROP EVENT TRIGGER IF EXISTS disallow_experimental_deps CASCADE;\\n\\\n        DROP EVENT TRIGGER IF EXISTS disallow_experimental_dependencies_on_views CASCADE;\\n\\\n        DROP FUNCTION IF EXISTS disallow_experimental_dependencies();\\n\\\n        DROP FUNCTION IF EXISTS disallow_experimental_view_dependencies();\\n\\\n        DROP FUNCTION IF EXISTS timescaledb_toolkit_probe;\"\n    )\n    .unwrap();\n\n    let lines = extension_file\n        .lines()\n        .map(|line| line.expect(\"cannot read install script\"))\n        .peekable();\n\n    let mut script_creator = UpdateScriptCreator {\n        lines,\n        upgrade_file,\n        new_stabilizations,\n    };\n\n    while script_creator.has_pending_input() {\n        let create = script_creator.find_create();\n        match create {\n            Some(Create::Function(create)) => {\n                script_creator.handle_create_functionlike(FunctionLike::Fn, create)\n            }\n            Some(Create::Aggregate(create)) => {\n                script_creator.handle_create_functionlike(FunctionLike::Agg, create)\n            }\n            Some(Create::Type(create)) => script_creator.handle_create_type(create),\n            Some(Create::Schema(create)) => {\n                // TODO is there something more principled to do here?\n                writeln!(script_creator.upgrade_file, \"CREATE SCHEMA {create}\").unwrap();\n            }\n            Some(Create::Operator(create)) => script_creator.handle_create_operator(create),\n            Some(Create::Cast(create)) => {\n                // TODO we don't have a stable one of these yet\n                // JOSH - we should probably check if the FUNCTION is experimental also\n                if create.contains(\"toolkit_experimental.\") || create.starts_with(\"(tests.\") {\n                    writeln!(script_creator.upgrade_file, \"CREATE CAST {create}\").unwrap();\n                    continue;\n                }\n                unimplemented!(\"unprepared for stable CAST: {create}\")\n            }\n            None => continue,\n        }\n    }\n}\n\nstruct UpdateScriptCreator<Lines, Dst>\nwhere\n    Lines: Iterator<Item = String>,\n    Dst: Write,\n{\n    lines: Peekable<Lines>,\n    upgrade_file: Dst,\n    new_stabilizations: StabilizationInfo,\n}\n\nenum Create {\n    Function(String),\n    Aggregate(String),\n    Type(String),\n    Operator(String),\n    Schema(String),\n    Cast(String),\n}\n\nconst MUST_FIND_MATCH: bool = false;\nconst ALLOW_NO_MATCH: bool = true;\n\nimpl<Lines, Dst> UpdateScriptCreator<Lines, Dst>\nwhere\n    Lines: Iterator<Item = String>,\n    Dst: Write,\n{\n    fn has_pending_input(&mut self) -> bool {\n        self.lines.peek().is_some()\n    }\n\n    // find a `CREATE <OBJECT KIND> <something>` and return the `<something>`\n    fn find_create(&mut self) -> Option<Create> {\n        for line in &mut self.lines {\n            // search for `CREATE FUNCTION/TYPE/OPERATOR <name>;`\n            let trimmed = line.trim_start();\n            if let Some(created) = trimmed.strip_prefix(\"CREATE \") {\n                let l = created.trim_start();\n                let create = match_start(\n                    l,\n                    [\n                        (\"FUNCTION\", &mut |l| Create::Function(l.to_string())),\n                        (\"AGGREGATE\", &mut |l| Create::Aggregate(l.to_string())),\n                        (\"TYPE\", &mut |l| Create::Type(l.to_string())),\n                        (\"OPERATOR\", &mut |l| Create::Operator(l.to_string())),\n                        (\"SCHEMA\", &mut |l| Create::Schema(l.to_string())),\n                        (\"CAST\", &mut |l| Create::Cast(l.to_string())),\n                    ],\n                );\n                if create.is_some() {\n                    return create;\n                }\n                unreachable!(\"unexpected CREATE `{trimmed}`\")\n            }\n\n            writeln!(self.upgrade_file, \"{line}\").unwrap();\n        }\n        return None;\n\n        // find which of a number of matchers a str starts with, and return the\n        // rest. In other words, if find the first matcher matcher such that the\n        // str is `<matcher> <remaining>` and return the `<remaining>`\n        #[allow(clippy::type_complexity)]\n        fn match_start<T, const N: usize>(\n            line: &str,\n            matchers: [(&str, &mut dyn FnMut(&str) -> T); N],\n        ) -> Option<T> {\n            for (matcher, constructor) in matchers {\n                if let Some(line) = line.strip_prefix(matcher) {\n                    let line = line.trim_start();\n                    return Some(constructor(line));\n                }\n            }\n            None\n        }\n    }\n\n    // handle a function-like create: if the function or aggregate is new in this\n    // version use `CREATE FUNCTION/AGGREGATE` to create the function, otherwise use\n    // `CREATE OR REPLACE` to update it to the newest version\n    fn handle_create_functionlike(&mut self, is_function: FunctionLike, mut create: String) {\n        if create.starts_with(\"toolkit_experimental\") || create.starts_with(\"tests\") {\n            writeln!(self.upgrade_file, \"{} {}\", is_function.create(), create).unwrap();\n            return;\n        }\n\n        if !create.contains(')') {\n            // look for the end of the argument list\n            create.push('\\n');\n            for line in &mut self.lines {\n                create.push_line(&line);\n                if line.contains(')') {\n                    break;\n                }\n            }\n        }\n\n        self.write_create_functionlike(is_function, &create);\n    }\n\n    fn write_create_functionlike(&mut self, is_function: FunctionLike, create_stmt: &str) {\n        // parse a function or aggregate\n        // it should look something like\n        // ```\n        // \"<function name>\"(\"<arg name>\" <arg type>,*) ...\n        // ```\n        let (name, rem) = parse_ident(create_stmt);\n        let types = parse_arg_types(rem);\n        let function = Function { name, types };\n\n        // write\n        if self.new_stabilizations.new_functions.contains(&function) {\n            writeln!(\n                self.upgrade_file,\n                \"{} {}\",\n                is_function.create(),\n                create_stmt\n            )\n            .expect(\"cannot write create function\")\n        } else {\n            writeln!(\n                self.upgrade_file,\n                \"{} {}\",\n                is_function.create_or_replace(),\n                create_stmt\n            )\n            .expect(\"cannot write create or replace function\")\n        }\n    }\n\n    fn handle_create_type(&mut self, create: String) {\n        let type_name = extract_name(&create);\n\n        if type_name.starts_with(\"toolkit_experimental\") || type_name.starts_with(\"tests\") {\n            writeln!(self.upgrade_file, \"CREATE TYPE {create}\").unwrap();\n            return;\n        }\n\n        if self.new_stabilizations.new_types.contains(&type_name) {\n            writeln!(self.upgrade_file, \"CREATE TYPE {create}\").unwrap();\n            return;\n        }\n\n        if create.trim_end().ends_with(';') {\n            // found `CREATE TYPE <name>;` we skip this in update scripts\n        } else if create.trim_end().ends_with('(') {\n            // found\n            // ```\n            // CREATE TYPE <name> (\n            //     ...\n            // );\n            // ```\n            // alter the type to match the new properties\n            let alters = self.get_alterable_properties();\n            self.write_alter_type(&type_name, &alters);\n        } else {\n            unreachable!()\n        }\n    }\n\n    fn get_alterable_properties(&mut self) -> Vec<Option<String>> {\n        self.get_properties(&ALTERABLE_PROPERTIES[..], ALLOW_NO_MATCH);\n        // Should return alters here, except PG12 doesn't allow alterations to type properties.\n        // Once we no longer support PG12 change this back to returning alters\n        vec![]\n    }\n\n    fn write_alter_type(&mut self, type_name: &str, alters: &[Option<String>]) {\n        let mut alter_statement = String::new();\n        for (i, alter) in alters.iter().enumerate() {\n            use std::fmt::Write;\n            let value = match alter {\n                None => continue,\n                Some(value) => value,\n            };\n            if alter_statement.is_empty() {\n                write!(&mut alter_statement, \"ALTER TYPE {type_name} SET (\")\n                    .expect(\"cannot write ALTER\");\n            } else {\n                alter_statement.push_str(\", \");\n            }\n            write!(\n                &mut alter_statement,\n                \"{} = {value}\",\n                ALTERABLE_PROPERTIES[i]\n            )\n            .expect(\"cannot write ALTER\");\n        }\n        if !alter_statement.is_empty() {\n            alter_statement.push_str(\");\");\n        }\n\n        writeln!(self.upgrade_file, \"{alter_statement}\").expect(\"cannot write ALTER TYPE\");\n    }\n\n    fn handle_create_operator(&mut self, create: String) {\n        assert!(create.trim_end().ends_with('('));\n        // found\n        // ```\n        // CREATE OPERATOR <op> (\n        //     PROCEDURE=...,\n        //     LEFTARG=...,\n        //     RIGHTARG=...\n        // );\n        // ```\n        // if any of `PROCEDURE`, `LEFTARG`, or `RIGHTARG` refer to and\n        // experimental object the operator is experimental, otherwise it isn't\n        let op = extract_name(&create);\n\n        let fields = self.get_properties(&[\"PROCEDURE\", \"LEFTARG\", \"RIGHTARG\"], MUST_FIND_MATCH);\n\n        let is_experimental = fields\n            .iter()\n            .filter_map(|f| f.as_ref())\n            .any(|f| f.contains(\"toolkit_experimental\"));\n\n        let parse_operator_arg_type = |field: &Option<String>| {\n            field\n                .as_ref()\n                .unwrap()\n                // remove everything after the comma, if one exists\n                .split_terminator(',')\n                .next()\n                .unwrap()\n                // remove any trailing comments\n                .split_terminator(\"/*\")\n                .next()\n                .unwrap()\n                .to_ascii_lowercase()\n                // handle `DOUBLE PRECISION`\n                .split_ascii_whitespace()\n                .map(|s| s.to_string())\n                .collect()\n        };\n        let operator = Function {\n            name: op.clone(),\n            types: vec![\n                parse_operator_arg_type(&fields[1]),\n                parse_operator_arg_type(&fields[2]),\n            ],\n        };\n        if is_experimental || self.new_stabilizations.new_operators.contains(&operator) {\n            writeln!(\n                self.upgrade_file,\n                \"CREATE OPERATOR {} (\\n    \\\n                    PROCEDURE={}\\n    \\\n                    LEFTARG={}\\n    \\\n                    RIGHTARG={}\\n    \\\n                );\",\n                op,\n                fields[0].as_ref().unwrap(),\n                fields[1].as_ref().unwrap(),\n                fields[2].as_ref().unwrap(),\n            )\n            .expect(\"cannot write CREATE OPERATOR\")\n        }\n    }\n\n    fn get_properties(&mut self, fields: &[&str], allow_no_match: bool) -> Vec<Option<String>> {\n        let mut properties = vec![None; fields.len()];\n        for line in &mut self.lines {\n            // found `)` means we're done with\n            // ```\n            // CREATE <object> <name> (\n            //     ...\n            // );\n            // ```\n            if line.trim_start().starts_with(')') {\n                break;\n            }\n            let mut split = line.split('=');\n            let field = split.next().unwrap().trim();\n            let value = split\n                .next()\n                .unwrap_or_else(|| panic!(\"no value for field {field}\"))\n                .trim();\n            assert_eq!(split.next(), None);\n\n            let mut found_match = false;\n            for (i, property) in fields.iter().enumerate() {\n                if field.eq_ignore_ascii_case(property) {\n                    properties[i] = Some(value.to_string());\n                    found_match = true;\n                }\n            }\n            if !found_match && !allow_no_match {\n                panic!(\"{field} is not considered an acceptable property for this object\")\n            }\n        }\n\n        properties\n    }\n}\n\nenum FunctionLike {\n    Fn,\n    Agg,\n}\n\nimpl FunctionLike {\n    fn create(&self) -> &'static str {\n        match self {\n            FunctionLike::Fn => \"CREATE FUNCTION\",\n            FunctionLike::Agg => \"CREATE AGGREGATE\",\n        }\n    }\n\n    fn create_or_replace(&self) -> &'static str {\n        match self {\n            FunctionLike::Fn => \"CREATE OR REPLACE FUNCTION\",\n            FunctionLike::Agg => \"CREATE OR REPLACE AGGREGATE\",\n        }\n    }\n}\n\nfn parse_arg_types(stmt: &str) -> Vec<Vec<String>> {\n    // extract the types from a\n    // `( <ident> <type segment>,* )`\n    // with arbitrary interior whitespace and comments into a\n    // `Vec<Vec<type segment>>`\n    let stmt = stmt.trim_start();\n    assert!(stmt.starts_with('('), \"stmt.starts_with('(') {stmt}\");\n    let end = stmt.find(')').expect(\"cannot find ')' for arg list\");\n    let args = &stmt[1..end];\n    let mut types = vec![];\n    // TODO strip out comments\n    for arg in args.split_terminator(',') {\n        let ty = arg\n            .split_whitespace()\n            .filter(remove_block_comments()) // skip any block comments\n            .skip(1) // skip the identifier at the start\n            .take_while(|s| !s.starts_with(\"--\")) // skip any line comments\n            .map(|s| s.to_ascii_lowercase())\n            .collect();\n\n        types.push(ty)\n    }\n    return types;\n\n    fn remove_block_comments() -> impl FnMut(&&str) -> bool {\n        let mut keep = true;\n        move |s| match *s {\n            \"*/\" => {\n                let ret = keep;\n                keep = true;\n                ret\n            }\n            \"/*\" => {\n                keep = false;\n                false\n            }\n            _ => keep,\n        }\n    }\n}\n\nfn parse_ident(mut stmt: &str) -> (String, &str) {\n    // parse `<ident>` or `\"<ident>\"`\n    let quoted = stmt.starts_with('\"');\n    if quoted {\n        stmt = &stmt[1..];\n        let end = stmt.find('\"').expect(\"cannot find closing quote\");\n        let ident = stmt[..end].to_string();\n        (ident, &stmt[end + 1..])\n    } else {\n        let end = stmt\n            .find(|c| !(char::is_alphanumeric(c) || c == '_'))\n            .expect(\"cannot find end of ident\");\n        let ident = stmt[..end].to_string();\n        (ident, &stmt[end..])\n    }\n}\n\nfn extract_name(line: &str) -> String {\n    let mut name: &str = line.split_ascii_whitespace().next().expect(\"no type name\");\n    if name.ends_with(';') {\n        name = &name[..name.len() - 1];\n    }\n    name.to_ascii_lowercase()\n}\n\n#[derive(Debug)]\n#[allow(dead_code)]\npub(crate) struct StabilizationInfo {\n    pub new_functions: HashSet<Function>,\n    pub new_types: HashSet<String>,\n    pub new_operators: HashSet<Function>,\n}\n\npub(crate) fn new_stabilizations(from_version: &str, to_version: &str) -> StabilizationInfo {\n    StabilizationInfo {\n        new_functions: stabilization_info::STABLE_FUNCTIONS(from_version, to_version),\n        new_types: stabilization_info::STABLE_TYPES(from_version, to_version),\n        new_operators: stabilization_info::STABLE_OPERATORS(from_version, to_version),\n    }\n}\n\n#[derive(Hash, Clone, PartialEq, Eq, Debug)]\npub(crate) struct Function {\n    name: String,\n    types: Vec<Vec<String>>,\n}\n\n#[derive(Debug)]\npub(crate) struct StaticFunction {\n    name: &'static str,\n    types: &'static [&'static [&'static str]],\n}\n\n#[derive(Eq, PartialEq, Ord, PartialOrd, Debug)]\nstruct Version {\n    major: u64,\n    minor: u64,\n    patch: u64,\n}\n\nfn version(s: &str) -> Version {\n    let mut nums = s.split('.');\n    let version = Version {\n        major: nums\n            .next()\n            .unwrap_or_else(|| panic!(\"no major version in `{s}`\"))\n            .parse()\n            .unwrap_or_else(|e| panic!(\"error {e} for major version in `{s}`\")),\n        minor: nums\n            .next()\n            .unwrap_or_else(|| panic!(\"no minor version in `{s}`\"))\n            .parse()\n            .unwrap_or_else(|e| panic!(\"error {e} for minor version in `{s}`\")),\n        patch: nums\n            .next()\n            .unwrap_or(\"0\")\n            .trim_end_matches(\"-dev\")\n            .parse()\n            .unwrap_or_else(|e| panic!(\"error {e} for major version in `{s}`\")),\n    };\n    if nums.next().is_some() {\n        panic!(\"extra `.`s in `{s}`\")\n    }\n    version\n}\n\nfn new_objects<'a, T: std::fmt::Debug>(\n    stabilizations: &'a [(&'a str, T)],\n    from_version: &'a str,\n    to_version: &'a str,\n) -> impl Iterator<Item = &'a (&'a str, T)> + 'a {\n    let to_version = to_version.trim_end_matches(\"-dev\");\n    let from_version = version(from_version);\n    let to_version = version(to_version);\n    stabilizations\n        .iter()\n        .skip_while(move |(version_str, _)| {\n            let version = version(version_str);\n            version > to_version\n        })\n        .take_while(move |(at, _)| at != &\"prehistory\" && version(at) > from_version)\n}\n\n#[macro_export]\nmacro_rules! functions_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($fn_name: ident ( $( $($fn_type: ident)+ ),* ) ),* $(,)?\n            }\n        )*\n    ) => {\n        #[allow(non_snake_case)]\n        pub(crate) fn $export_symbol(from_version: &str, to_version: &str) -> super::HashSet<super::Function> {\n            use super::*;\n            static STABILIZATIONS: &[(&str, &[StaticFunction])] = &[\n                $(\n                    (\n                        $version,\n                        &[\n                            $(StaticFunction {\n                                name: stringify!($fn_name),\n                                types: &[$(\n                                    &[$(\n                                        stringify!($fn_type),\n                                    )*],\n                                )*],\n                            },)*\n                        ],\n                    ),\n                )*\n            ];\n\n            new_objects(STABILIZATIONS, from_version, to_version)\n                .flat_map(|(_, creates)| creates.into_iter().map(|StaticFunction { name, types }|\n                    Function {\n                        name: name.to_ascii_lowercase(),\n                        types: types.into_iter().map(|v|\n                                v.into_iter().map(|s| s.to_ascii_lowercase()).collect()\n                            ).collect(),\n                    })\n                )\n                .collect()\n        }\n    };\n}\n\n#[macro_export]\nmacro_rules! types_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($type_name: ident),* $(,)?\n            }\n        )*\n    ) => {\n        #[allow(non_snake_case)]\n        pub(crate) fn $export_symbol(from_version: &str, to_version: &str) -> super::HashSet<String> {\n            use super::*;\n            static STABILIZATIONS: &[(&str, &[&str])] = &[\n                $(\n                    (\n                        $version,\n                        &[\n                            $(stringify!($type_name),)*\n                        ],\n                    ),\n                )*\n            ];\n\n            new_objects(STABILIZATIONS, from_version, to_version)\n                .flat_map(|(_, creates)| creates.into_iter().map(|t| t.to_ascii_lowercase()) )\n                .collect()\n        }\n    };\n}\n\n#[macro_export]\nmacro_rules! operators_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($operator_name: literal ( $( $($fn_type: ident)+ ),* ) ),* $(,)?\n            }\n        )*\n    ) => {\n        #[allow(non_snake_case)]\n        pub(crate) fn $export_symbol(from_version: &str, to_version: &str) -> super::HashSet<super::Function> {\n            use super::*;\n            static STABILIZATIONS: &[(&str, &[StaticFunction])] = &[\n                $(\n                    (\n                        $version,\n                        &[\n                            $(StaticFunction {\n                                name: $operator_name,\n                                types: &[$(\n                                    &[$(\n                                        stringify!($fn_type),\n                                    )*],\n                                )*],\n                            },)*\n                        ],\n                    ),\n                )*\n            ];\n\n            new_objects(STABILIZATIONS, from_version, to_version)\n                .flat_map(|(_, creates)| creates.into_iter().map(|StaticFunction { name, types }|\n                    Function {\n                        name: name.to_ascii_lowercase(),\n                        types: types.into_iter().map(|v|\n                                v.into_iter().map(|s| s.to_ascii_lowercase()).collect()\n                            ).collect(),\n                    })\n                )\n                .collect()\n        }\n    };\n}\n"
  },
  {
    "path": "tools/release",
    "content": "#!/bin/sh\n\n# This script automates release creation:\n# 1. Create release branch from target commit.\n# 1a. Validate contents of target commit (just upgradeable_from currently).\n# 2. Set toolkit version on branch.\n# 3. Run tests.\n# 4. Push (if -push) the branch so release-build-scripts repository [1] can see the commit from #2.\n# 5. Trigger (if -push) toolkit packaging actions in release-build-scripts repository.\n# 6. Tag the release (and push, if -push). [2]\n# 7. Prepare the main branch for the next release cycle.\n# 7a. Update upgradeable_form in control file.\n# 7b. Set toolkit version to released version with '-dev' appended.\n# 7c. Update Changelog.md .\n# 7d. Push to and create pull request for post-$VERSION branch (if -push).\n# 8. File issue for release tasks that are not yet automated (if -push).\n\n# [1] We need a self-hosted runner for arm64 build, which we can only get with\n#     a private repository, so we must delegate packaging to that.\n\n# [2] This means we publish a tag before testing binaries.  We'd rather test first.\n#     TODO How?\n#     - Can we have release-build-scripts gh back to an action over here?\n#     - Can we have a trigger that watches for release-build-scripts action to finish?\n\n# Sample run:\n# tools/release -n -push -version 1.11.0 9c2b04d\n\n# git commit records these on commits (yes, all three).\n# TODO What should we use?  I pulled this from the deb package metadata\nEMAIL=hello@timescale.com\nGIT_AUTHOR_NAME=tools/release\nGIT_COMMITTER_NAME=$GIT_AUTHOR_NAME\nexport EMAIL GIT_AUTHOR_NAME GIT_COMMITTER_NAME\n\nMAIN_BRANCH=main\nBRANCH_BASENAME=forge-stable-\nCONTROL=extension/timescaledb_toolkit.control\nTOML=extension/Cargo.toml\nUPGRADEABLE_FROM_RE=\"^# upgradeable_from = '[^']*'\\$\"\nNEXT_RELEASE_RE='^## Next Release (Date TBD)'\n\n. tools/dependencies.sh\n\nset -ex\n\n# TODO Install these into timescaledev/toolkit-builder image and delete this block.\nif [ \"$1\" = setup ]; then\n    # Install cargo set-version (and cargo install is not idempotent).\n    if ! cargo help set-version > /dev/null; then\n        cargo install --version =$CARGO_EDIT cargo-edit\n    fi\n    # Install gh\n    gh=`basename $GH_DEB_URL`\n    curl --fail -LO $GH_DEB_URL\n    sha256sum -c - <<EOF\n$GH_DEB_SHA256  $gh\nEOF\n    sudo dpkg -i $gh\n    exit\nfi\n\nprint() {\n    printf '%s\\n' \"$*\"\n}\n\ndie() {\n    st=${?:-0}\n    if [ $st -eq 0 ]; then\n        st=2\n    fi\n    print \"$*\" >&2\n    exit $st\n}\n\nusage() {\n    die 'release [-n] [-push] -version VERSION COMMIT'\n}\n\n# Return 0 iff working directory is clean.\n# Also prints any diff.\nassert_clean() {\n    $nop git diff --exit-code\n}\n\n# Return 0 iff working directory is dirty.\n# Also prints any diff.\nassert_dirty() {\n    [ -n \"$nop\" ] && return\n    ! assert_clean\n}\n\n# Use start_commit, commit, and finish_commit to safely build a commit from\n# multiple automated edits.\n# - start_commit [file names]\n#   Start a commit with the named changed files.\n#   Any other edited file (dirty directory after commit) is an error.\n# - commit [file names]\n#   Amend the commit after each automated edit.\n#   Any other edited file (dirty directory after commit) is an error.\n# - finish_commit MESSAGE\n#   Finalize the commit with the commit message MESSAGE.\n#   Any edited files is an error.\nstart_commit() {\n    [ -z \"$_PENDING_COMMIT\" ] || die 'BUG: start_commit called twice'\n    _PENDING_COMMIT=1\n    $nop git add \"$@\"\n    $nop git commit -m pending\n    assert_clean || die \"working directory should be clean after commit $@\"\n}\n\ncommit() {\n    [ -n \"$_PENDING_COMMIT\" ] || die 'BUG: commit called without start_commit'\n    $nop git add \"$@\"\n    $nop git commit --no-edit --amend\n    assert_clean || die \"working directory should be clean after commit $@\"\n}\n\nfinish_commit() {\n    [ -n \"$_PENDING_COMMIT\" ] || die 'BUG: finish_commit called without start_commit'\n    assert_clean || die \"working directory should be clean to finish commit '$1'\"\n    _PENDING_COMMIT=\n    (export GIT_COMMITTER_DATE=\"`date`\" && $nop git commit --no-edit --amend \"--date=$GIT_COMMITTER_DATE\" -m \"$1\")\n}\n\n# Return 0 if this is a minor release (i.e. $PATCH is greater than zero).\nrelease_is_minor() {\n    [ $PATCH -eq 0 ]\n}\n\n# Super simple option processing.\npush=false\nwhile [ $# -gt 0 ]; do\n    arg=$1\n    shift\n    case \"$arg\" in\n        -n)\n            dry_run_flag=--dry-run\n            nop=:\n            ;;\n\n        # TODO Remove -y alias for -push .\n        -push | -y)\n            push=true\n            ;;\n\n        -version)\n            VERSION=$1\n            shift\n            COMMIT=$1\n            shift\n            ;;\n\n        *)\n            usage\n            ;;\n    esac\ndone\n\n[ -n \"$VERSION\" ] && [ -n \"$COMMIT\" ] || usage\n\n# And away we go!\n\nMAJOR=${VERSION%%.*}\nminpat=${VERSION#*.}\nMINOR=${minpat%.*}\nPATCH=${minpat#*.}\n\nPOST_REL_BRANCH=post-$VERSION\n\n# 0. Sanity-check the surroundings.\n# working directory clean?\nassert_clean || die 'cowardly refusing to operate on dirty working directory'\n\n# 1. Create release branch from target commit.\nbranch=\"$BRANCH_BASENAME\"$VERSION\n$nop git checkout -b $branch $COMMIT\n\n# Sanity-check the branch contents.\n# control file matches expectations?\ncount=`grep -c \"$UPGRADEABLE_FROM_RE\" $CONTROL` || die \"upgradeable_from line malformed\"\nif [ \"$count\" -ne 1 ]; then\n    print >&2 \"too many upgradeable_from lines matched:\"\n    grep >&2 \"$UPGRADEABLE_FROM_RE\" $CONTROL\n    die\nfi\n# If we forget to update the Changelog (or forget to cherry-pick Changelog\n# updates), show a clear error message rather than letting the ed script fail\n# mysteriously.\ngrep -qs \"$NEXT_RELEASE_RE\" Changelog.md || die 'Changelod.md lacks \"Next Release\" section'\n\n# 1a. Validate contents of target commit (just upgradeable_from currently).\nif ! release_is_minor; then\n    # Releasing e.g. 1.13.2 - this one might be a cherry-pick, so we need to ensure upgradeable from 1.13.1 .\n    # It is conceivable that we could intend to release 1.17.1 without\n    # allowing upgrade from 1.17.0, but we can cross that bridge if we come\n    # to it.\n    prev=$MAJOR.$MINOR.$(( PATCH - 1 ))\n    # The set of lines matching this pattern is a subset of the set required in preflight above.\n    grep -Eqs \"^# upgradeable_from = '[^']*,?$prev[,']\" $CONTROL || die \"$prev missing from upgradeable_from \"\nfi\n# Else releasing e.g. 1.13.0 - these are never cherrypicks and we automatically set upgradeable_from on main.\n\n# 2. Set toolkit version.\ncargo set-version $dry_run_flag -p timescaledb_toolkit $VERSION\nassert_dirty || die \"failed to set toolkit version to $VERSION in $TOML\"\nstart_commit $TOML\n# Update cargo.lock - this form of cargo update doesn't update dependency versions.\n$nop cargo update -p timescaledb_toolkit\nassert_dirty || die \"failed to set toolkit version to $VERSION in Cargo.lock\"\ncommit Cargo.lock\n# Update Changelog.md .\nbranch_commit_date=`git log -1 --pretty=format:%as $branch_commit`\n$nop ed Changelog.md <<EOF\n/$NEXT_RELEASE_RE/\nd\ni\n## [$VERSION](https://github.com/timescale/timescaledb-toolkit/releases/tag/$VERSION) ($branch_commit_date)\n.\nwq\nEOF\nassert_dirty || die 'failed to update Changelog.md for next release'\ncommit Changelog.md\nfinish_commit \"release $VERSION\"\n$nop git show\n\n# 3. Run tests.\nfor pg in $PG_VERSIONS; do\n    $nop tools/build -pg$pg test-extension\n    $nop rm -rf target # we only have limited disk space on github's runner\ndone\nassert_clean || die 'tools/build should not dirty the working directory'\n\n# 4. Push the branch\nif $push; then\n    $nop git push origin $branch\nfi\n\n# 5. Trigger toolkit packaging actions in release-build-scripts repository.\nbranch_commit=`git log -1 --pretty=format:%h`\nif $push; then\n    $nop gh workflow run toolkit-package.yml \\\n        -R timescale/release-build-scripts \\\n        -r $MAIN_BRANCH \\\n        -f version=$VERSION \\\n        -f commit-id=$branch_commit \\\n        -f upload-artifacts=true\nfi\n\n# 6. Tag the release.\n$nop git tag $VERSION\nif $push; then\n    $nop git push origin $VERSION\n    # TODO gh release\n#     ed -s > release-notes Changelog.md <<EOF\n# /^## \\[[^]]*](https:..github.com.timescale.timescaledb-toolkit.releases.tag/\n# +,/^## \\[[^]]*](https:..github.com.timescale.timescaledb-toolkit.releases.tag/p\n# EOF\n#     gh release create -dF release-notes --target $VERSION $VERSION\nfi\n\n# 7. Prepare the main branch for the next release cycle.\n# Github action gives us a shallow checkout which we must deepen before we can push changes.\n$nop git fetch --deepen=2147483647 origin $MAIN_BRANCH\n$nop git checkout -b $POST_REL_BRANCH $MAIN_BRANCH\n\n# 7a. Update upgradeable_form in control file.\n$nop sed --in-place \"/$UPGRADEABLE_FROM_RE/ { s/'\\$/, $VERSION'/ }\" $CONTROL\nassert_dirty || die \"failed to update $CONTROL for next release\"\nstart_commit $CONTROL\n\nif release_is_minor; then\n    # 7b. Set toolkit version to released version with '-dev' appended.\n    # Skip for patch releases:  we've already started the next minor version in that case.\n    DEV_VERSION=$MAJOR.$(( MINOR + 1 )).0-dev\n    cargo set-version $dry_run_flag -p timescaledb_toolkit $DEV_VERSION\n    assert_dirty || die \"failed to set toolkit version to $DEV_VERSION in $TOML\"\n    commit $TOML\n    # Update cargo.lock - this form of cargo update doesn't update dependency versions.\n    $nop cargo update -p timescaledb_toolkit\n    assert_dirty || die \"failed to set toolkit version to $DEV_VERSION in Cargo.lock\"\n    commit Cargo.lock\n\n    # 7c. Update Changelog.md .\n    # Skip for patch releases as it's not clear how to automate the cherry-pick case.\n    # For now, we just have to add patch releases to the main Changelog manually.\n    # The edit we apply here for minor releases would be wrong in the\n    # cherry-pick case, as it would erroneously list the skipped changes on\n    # main as part of the patch release.  This script has no way to\n    # distinguish blocks of text belonging to one release from another, so\n    # automating that case is probably not feasible.\n    # TODO Or is it?\n    branch_commit_date=`git log -1 --pretty=format:%as $branch_commit`\n    $nop ed Changelog.md <<EOF\n/$NEXT_RELEASE_RE/\na\n\n#### New experimental features\n\n#### Bug fixes\n\n#### Other notable changes\n\n#### Shout-outs\n\n**Full Changelog**: [TODO]\n\n## [$VERSION](https://github.com/timescale/timescaledb-toolkit/releases/tag/$VERSION) ($branch_commit_date)\n.\nwq\nEOF\n    assert_dirty || die 'failed to update Changelog.md for next release'\n    commit Changelog.md\n\n    finish_commit \"start $DEV_VERSION\"\nelse\n    finish_commit \"add $VERSION to upgradeable_from\"\nfi\n$nop git show\n\n# We've had a lot of trouble with the rest of these, so let's continue on\n# error, to see what works and what doesn't.\nset +e\n# TODO Carefully attempt to report errors but keep going on all steps after we push the tag in step 6.\n\nif $push; then\n    # 7d. Push to $POST_REL_BRANCH branch.\n    $nop git push origin HEAD:$POST_REL_BRANCH\n\n    # Run these next steps as github-actions[bot]\n    GITHUB_TOKEN=\"$ACTIONS_GITHUB_TOKEN\"\n\n    $nop gh pr create -R timescale/timescaledb-toolkit -B $MAIN_BRANCH --fill -H $POST_REL_BRANCH\n\n    # 8. File issue for release tasks that are not yet automated.\n    $nop gh issue create -R timescale/timescaledb-toolkit -F- -t \"Release $VERSION\" <<EOF\n- [ ] Docker HA image\n\n[Sample pull request](https://github.com/timescale/timescaledb-docker-ha/pull/298)\n\nAdd new version to \\`TIMESCALEDB_TOOLKIT_EXTENSIONS\\` in:\n- \\`.github/workflows/build_image.yaml\\`\n- \\`.github/workflows/publish_image.yaml\\`\n- \\`Makefile\\`\n\n- [ ] hot-forge\n\n[Sample pull request](https://github.com/timescale/hot-forge/pull/67)\n\nAdd two new lines to \\`bundles.yaml\\` containing the new version tag:\n\\`\\`\\`\n- repository: https://github.com/timescale/timescaledb-toolkit\n  tag: $VERSION\n\\`\\`\\`\n\nAnd update \\`.github/build_bundles.py\\` if new pgrx is required.\n\n- [ ] Copy Changelog.md entries for this release into\n  [github release](https://github.com/timescale/timescaledb-toolkit/releases/tag/$VERSION)\n\n- [ ] Update Homebrew\n\nBuild binaries on multiple Mac versions/architectures and submit a pull request like\n[this example](https://github.com/timescale/homebrew-tap/pull/29/files)\nEOF\nfi\n"
  },
  {
    "path": "tools/sql-doctester/Cargo.toml",
    "content": "[package]\nname = \"sql-doctester\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nbytecount = \"0.6.2\"\nclap = { version = \"3.2.15\", features = [\"wrap_help\"] }\ncolored = \"2.0.0\"\npostgres = \"=0.19.10\" # pinned because we pin tokio-postgres\npulldown-cmark = \"0.8.0\"\nrayon = \"1.5\"\ntokio-postgres = \"=0.7.13\" # pinned because 0.7.11 added `SimpleQueryMessage::RowDescription`\nuuid = { version = \"0.8\", features = [\"v4\"] }\nwalkdir = \"2\"\n"
  },
  {
    "path": "tools/sql-doctester/Readme.md",
    "content": "## SQL Doctester ##\n\nTest SQL code in Markdown files.\n\nThis tool looks through a directory for markdown files containing SQL, runs any\nexamples it finds, and validates its output. For instance, when run against this\nfile it will validate that the following SQL runs correctly:\n\n```SQL\nSELECT count(v), sum(v), avg(v) FROM generate_series(1, 10) v;\n```\n```output\n count | sum |                avg\n ------+-----+--------------------\n    10 |  55 | 5.5000000000000000\n```\n\nIf we were to have errors in the example, for instance, we `count` and `sum`\nswapped, the tool will report the errors, and where the output differs from the\nexpected output, like so:\n\n```\nTests Failed\n\nReadme.md:9 `SQL Doctester`\n\nSELECT count(v), sum(v), avg(v) FROM generate_series(1, 10) v;\n\nError: output has a different values than expected.\nExpected\n55 | 10 | 5.5000000000000000\n(1 rows)\n\nReceived\n10 | 55 | 5.5000000000000000\n(1 rows)\n\nDelta\n-55+10 | -10+55 | 5.5000000000000000\n```\n\n## Installation ##\n\n```bash\ncargo install --git https://github.com/timescale/timescaledb-toolkit.git --branch main sql-doctester\n```\n\n## Usage ##\n\n```\nsql-doctester\n\nUSAGE:\n    sql-doctester [OPTIONS] <tests>\n\nFLAGS:\n        --help       Prints help information\n    -V, --version    Prints version information\n\nOPTIONS:\n    -d, --database <database>\n            postgres database the root connection should use. By default this DB will only be used\n            to spawn the individual test databases; no tests will run against it.\n    -h, --host <hostname>                    postgres host\n    -a, --password <password>                postgres password\n    -p, --port <portnumber>                  postgres port\n    -f, --startup-file <startup_file>\n            File containing SQL commands that should be run when each test database is created.\n\n    -s, --startup-script <startup_script>\n            SQL command that should be run when each test database is created.\n\n    -u, --user <username>                    postgres user\n\nARGS:\n    <tests>    Path in which to search for tests\n```\n\n## Formatting ##\n\nThe tool looks through every markdown file in the provided path for SQL code\nblocks like\n\n    ```SQL\n    SELECT column_1, column_2, etc FROM foo\n    ```\n\nand will try to run them. The SQL is assumed to be followed with an `output`\nblock like which contains the expected output for the command\n\n    ```output\n     column 1 | column 2 | etc\n    ----------+----------+-----\n      value 1 |  value 1 | etc\n    ```\n\nOnly the actual values are checked; the header, along with leading and trailing\nwhitespace are ignored. If no `output` is provided the tester will validate that\nthe output should be empty.\n\nOutput validation can be suppressed by adding `ignore-output` after\nthe `SQL` tag, like so\n\n    ```SQL,ignore-output\n    SELECT non_validated FROM foo\n    ```\nin which case the SQL will be run, and its output ignored.\n\nSQL code blocks can be skipped entirely be adding `ignore` after the tag as in\n\n    ```SQL,ignore\n    This never runs, so it doesn't matter if it's valid SQL\n    ```\n\nBy default, each code block is run in its own transaction, which is rolled back\nafter the command completes. If you want to run outside a transaction, because\nyou're running commands that cannot be run within a transaction, or because you\nwant to change global state, you can mark a block as non-transactional like so\n\n    ```SQL,non-transactional\n    CREATE TABLE bar();\n    ```\n\nEvery file is run in its own database, so such commands can only affect the\nremainder of the current file. This tag can be combined with any of the others.\n\nThe tool supports adding startup scripts that are run first on every new\ndatabase. This can be useful for repetitive initialization tasks, like CREATEing\nextensions that must be done for every file. For file-specific initialization,\nyou can you `non-transactional` blocks. These blocks can be hidden `<div>` like\nso\n\n    <div hidden>\n\n    ```SQL,non-transactional\n    CREATE TABLE data()\n    ```\n\n    </div>\n\nif you want them to be invisible to readers.\n\n## Acknowledgements ##\n\nInspired by [rustdoc](https://doc.rust-lang.org/rustdoc/what-is-rustdoc.html)\nand [rust-skeptic](https://github.com/budziq/rust-skeptic).\n"
  },
  {
    "path": "tools/sql-doctester/src/main.rs",
    "content": "use std::{\n    collections::HashMap,\n    ffi::OsStr,\n    fs,\n    io::{self, Write},\n    process::exit,\n};\n\nuse colored::Colorize;\n\nuse clap::{Arg, Command};\nuse runner::ConnectionConfig;\nmod parser;\nmod runner;\n\nfn main() {\n    let matches = Command::new(\"sql-doctester\")\n        .about(\"Runs sql commands from docs/ dir to test out toolkit\")\n        .arg_required_else_help(true)\n        .arg(Arg::new(\"HOST\").short('h').long(\"host\").takes_value(true))\n        .arg(Arg::new(\"PORT\").short('p').long(\"port\").takes_value(true))\n        .arg(Arg::new(\"USER\").short('u').long(\"user\").takes_value(true))\n        .arg(\n            Arg::new(\"PASSWORD\")\n                .short('a')\n                .long(\"password\")\n                .takes_value(true),\n        )\n        .arg(Arg::new(\"DB\").short('d').long(\"database\").takes_value(true))\n        .arg(Arg::new(\"INPUT\").takes_value(true))\n        .mut_arg(\"help\", |_h| Arg::new(\"help\").long(\"help\"))\n        .get_matches();\n\n    let dirname = matches.value_of(\"INPUT\").expect(\"need input\");\n\n    let connection_config = ConnectionConfig {\n        host: matches.value_of(\"HOST\"),\n        port: matches.value_of(\"PORT\"),\n        user: matches.value_of(\"USER\"),\n        password: matches.value_of(\"PASSWORD\"),\n        database: matches.value_of(\"DB\"),\n    };\n\n    let startup_script = include_str!(\"startup.sql\");\n\n    let all_tests = extract_tests(dirname);\n\n    let mut num_errors = 0;\n    let stdout = io::stdout();\n    let mut out = stdout.lock();\n\n    let on_error = |test: Test, error: runner::TestError| {\n        if num_errors == 0 {\n            let _ = writeln!(&mut out, \"{}\\n\", \"Tests Failed\".bold().red());\n        }\n        num_errors += 1;\n        let _ = writeln!(\n            &mut out,\n            \"{} {}\\n\",\n            test.location.bold().blue(),\n            test.header.bold().dimmed()\n        );\n        let _ = writeln!(&mut out, \"{}\", error.annotate_position(&test.text));\n        let _ = writeln!(&mut out, \"{error}\\n\");\n    };\n\n    runner::run_tests(connection_config, startup_script, all_tests, on_error);\n    if num_errors > 0 {\n        exit(1)\n    }\n    let _ = writeln!(&mut out, \"{}\\n\", \"Tests Passed\".bold().green());\n}\n\n#[derive(Debug, PartialEq, Eq)]\n#[must_use]\npub struct TestFile {\n    name: String,\n    stateless: bool,\n    tests: Vec<Test>,\n}\n\n#[derive(Debug, PartialEq, Eq)]\n#[must_use]\npub struct Test {\n    location: String,\n    header: String,\n    text: String,\n    output: Vec<Vec<String>>,\n    transactional: bool,\n    ignore_output: bool,\n    precision_limits: HashMap<usize, usize>,\n}\n\nfn extract_tests(root: &str) -> Vec<TestFile> {\n    // TODO handle when root is a file\n    let mut all_tests = vec![];\n    let walker = walkdir::WalkDir::new(root)\n        .follow_links(true)\n        .sort_by(|a, b| a.path().cmp(b.path()));\n    for entry in walker {\n        let entry = entry.unwrap();\n        if !entry.file_type().is_file() {\n            continue;\n        }\n\n        if entry.path().extension() != Some(OsStr::new(\"md\")) {\n            continue;\n        }\n\n        let realpath;\n        let path = if entry.file_type().is_symlink() {\n            realpath = fs::read_link(entry.path()).unwrap();\n            &*realpath\n        } else {\n            entry.path()\n        };\n        let contents = fs::read_to_string(path).unwrap();\n\n        let tests = parser::extract_tests_from_string(&contents, &entry.path().to_string_lossy());\n        if !tests.tests.is_empty() {\n            all_tests.push(tests)\n        }\n    }\n    all_tests\n}\n"
  },
  {
    "path": "tools/sql-doctester/src/parser.rs",
    "content": "use std::collections::HashMap;\n\nuse pulldown_cmark::{\n    CodeBlockKind::Fenced,\n    CowStr, Event, Parser,\n    Tag::{CodeBlock, Heading},\n};\n\nuse crate::{Test, TestFile};\n\n// parsers the grammar `(heading* (test output?)*)*`\npub fn extract_tests_from_string(s: &str, file_stem: &str) -> TestFile {\n    let mut parser = Parser::new(s).into_offset_iter().peekable();\n    let mut heading_stack = vec![];\n    let mut tests = vec![];\n\n    let mut last_test_seen_at = 0;\n    let mut lines_seen = 0;\n\n    let mut stateless = true;\n\n    // consume the parser until an tag is reached, performing an action on each text\n    macro_rules! consume_text_until {\n        ($parser: ident yields $end: pat => $action: expr) => {\n            for (event, _) in &mut parser {\n                match event {\n                    Event::Text(text) => $action(text),\n                    $end => break,\n                    _ => (),\n                }\n            }\n        };\n    }\n\n    'block_hunt: while let Some((event, span)) = parser.next() {\n        match event {\n            // we found a heading, add it to the stack\n            Event::Start(Heading(level)) => {\n                heading_stack.truncate(level as usize - 1);\n                let mut header = \"`\".to_string();\n                consume_text_until!(parser yields Event::End(Heading(..)) =>\n                    |text: CowStr| header.push_str(&text)\n                );\n                header.truncate(header.trim_end().len());\n                header.push('`');\n                heading_stack.push(header);\n            }\n\n            // we found a code block, if it's a test add the test\n            Event::Start(CodeBlock(Fenced(ref info))) => {\n                let code_block_info = parse_code_block_info(info);\n\n                // non-test code block, consume it and continue looking\n                if let BlockKind::Other = code_block_info.kind {\n                    for (event, _) in &mut parser {\n                        if let Event::End(CodeBlock(Fenced(..))) = event {\n                            break;\n                        }\n                    }\n                    continue 'block_hunt;\n                }\n\n                let current_line = {\n                    let offset = span.start;\n                    lines_seen += bytecount::count(&s.as_bytes()[last_test_seen_at..offset], b'\\n');\n                    last_test_seen_at = offset;\n                    lines_seen + 1\n                };\n\n                if let BlockKind::Output = code_block_info.kind {\n                    panic!(\n                        \"found output with no test test.\\n{file_stem}:{current_line} {heading_stack:?}\"\n                    )\n                }\n\n                assert!(matches!(code_block_info.kind, BlockKind::Sql));\n\n                stateless &= code_block_info.transactional;\n                let mut test = Test {\n                    location: format!(\"{file_stem}:{current_line}\"),\n                    header: if heading_stack.is_empty() {\n                        \"<root>\".to_string()\n                    } else {\n                        heading_stack.join(\"::\")\n                    },\n                    text: String::new(),\n                    output: Vec::new(),\n                    transactional: code_block_info.transactional,\n                    ignore_output: code_block_info.ignore_output,\n                    precision_limits: code_block_info.precision_limits,\n                };\n\n                // consume the lines of the test\n                consume_text_until!(parser yields Event::End(CodeBlock(Fenced(..))) =>\n                    |text: CowStr| test.text.push_str(&text)\n                );\n\n                // search to see if we have output\n                loop {\n                    match parser.peek() {\n                        // we found a code block, is it output?\n                        Some((Event::Start(CodeBlock(Fenced(info))), _)) => {\n                            let code_block_info = parse_code_block_info(info);\n                            match code_block_info.kind {\n                                // non-output, continue at the top\n                                BlockKind::Sql | BlockKind::Other => {\n                                    tests.push(test);\n                                    continue 'block_hunt;\n                                }\n\n                                // output, consume it\n                                BlockKind::Output => {\n                                    if !test.precision_limits.is_empty()\n                                        && !code_block_info.precision_limits.is_empty()\n                                    {\n                                        panic!(\n                                            \"cannot have precision limits on both test and output.\\n{file_stem}:{current_line} {heading_stack:?}\"\n                                        )\n                                    }\n                                    test.precision_limits = code_block_info.precision_limits;\n                                    let _ = parser.next();\n                                    break;\n                                }\n                            }\n                        }\n\n                        // test must be over, continue at the top\n                        Some((Event::Start(CodeBlock(..)), _))\n                        | Some((Event::Start(Heading(..)), _)) => {\n                            tests.push(test);\n                            continue 'block_hunt;\n                        }\n\n                        // EOF, we're done\n                        None => {\n                            tests.push(test);\n                            break 'block_hunt;\n                        }\n\n                        // for now we allow text between the test and it's output\n                        // TODO should/can we forbid this?\n                        _ => {\n                            let _ = parser.next();\n                        }\n                    };\n                }\n\n                // consume the output\n                consume_text_until!(parser yields Event::End(CodeBlock(Fenced(..))) =>\n                    |text: CowStr| {\n                        let rows = text.split('\\n').skip(2).filter(|s| !s.is_empty()).map(|s|\n                            s.split('|').map(|s| s.trim().to_string()).collect::<Vec<_>>()\n                        );\n                        test.output.extend(rows);\n                    }\n                );\n\n                tests.push(test);\n            }\n\n            _ => (),\n        }\n    }\n    TestFile {\n        name: file_stem.to_string(),\n        stateless,\n        tests,\n    }\n}\n\nstruct CodeBlockInfo {\n    kind: BlockKind,\n    transactional: bool,\n    ignore_output: bool,\n    precision_limits: HashMap<usize, usize>,\n}\n\n#[derive(Clone, Copy)]\nenum BlockKind {\n    Sql,\n    Output,\n    Other,\n}\n\nfn parse_code_block_info(info: &str) -> CodeBlockInfo {\n    let tokens = info.split(',');\n\n    let mut info = CodeBlockInfo {\n        kind: BlockKind::Other,\n        transactional: true,\n        ignore_output: false,\n        precision_limits: HashMap::new(),\n    };\n\n    for token in tokens {\n        match token.trim() {\n            \"ignore\" => {\n                if let BlockKind::Sql = info.kind {\n                    info.kind = BlockKind::Other;\n                }\n            }\n            \"non-transactional\" => info.transactional = false,\n            \"ignore-output\" => info.ignore_output = true,\n            \"output\" => info.kind = BlockKind::Output,\n            s if s.eq_ignore_ascii_case(\"sql\") => info.kind = BlockKind::Sql,\n            p if p.starts_with(\"precision\") => {\n                // syntax `precision(col: bytes)`\n                let precision_err =\n                    || -> ! { panic!(\"invalid syntax for `precision(col: bytes)` found `{p}`\") };\n                let arg = &p[\"precision\".len()..];\n                if arg.as_bytes().first() != Some(&b'(') || arg.as_bytes().last() != Some(&b')') {\n                    precision_err()\n                }\n                let arg = &arg[1..arg.len() - 1];\n                let args: Vec<_> = arg.split(':').collect();\n                if args.len() != 2 {\n                    precision_err()\n                }\n                let column = args[0].trim().parse().unwrap_or_else(|_| precision_err());\n                let length = args[1].trim().parse().unwrap_or_else(|_| precision_err());\n                let old = info.precision_limits.insert(column, length);\n                if old.is_some() {\n                    panic!(\"duplicate precision for column {column}\")\n                }\n            }\n            _ => {}\n        }\n    }\n\n    info\n}\n\n#[cfg(test)]\nmod test {\n    use std::collections::HashMap;\n\n    #[test]\n    fn extract() {\n        use super::{Test, TestFile};\n\n        let file = r##\"\n# Test Parsing\n```SQL\nselect * from foo\n```\n```output\n```\n\n```SQL\nselect * from multiline\n```\n```output\n ?column?\n----------\n    value\n```\n\n## ignored\n```SQL,ignore\nselect * from foo\n```\n\n## non-transactional\n```SQL,non-transactional\nselect * from bar\n```\n```output, precision(1: 3)\n a | b\n---+---\n 1 | 2\n```\n\n## no output\n```SQL,ignore-output\nselect * from baz\n```\n\n## end by header\n```SQL\nselect * from quz\n```\n\n## end by file\n```SQL\nselect * from qat\n```\n\"##;\n\n        let tests = super::extract_tests_from_string(file, \"/test/file.md\");\n        let expected = TestFile {\n            name: \"/test/file.md\".to_string(),\n            stateless: false,\n            tests: vec![\n                Test {\n                    location: \"/test/file.md:3\".to_string(),\n                    header: \"`Test Parsing`\".to_string(),\n                    text: \"select * from foo\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                },\n                Test {\n                    location: \"/test/file.md:9\".to_string(),\n                    header: \"`Test Parsing`\".to_string(),\n                    text: \"select * from multiline\\n\".to_string(),\n                    output: vec![vec![\"value\".to_string()]],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                },\n                Test {\n                    location: \"/test/file.md:24\".to_string(),\n                    header: \"`Test Parsing`::`non-transactional`\".to_string(),\n                    text: \"select * from bar\\n\".to_string(),\n                    output: vec![vec![\"1\".to_string(), \"2\".to_string()]],\n                    transactional: false,\n                    ignore_output: false,\n                    precision_limits: [(1, 3)].iter().cloned().collect(),\n                },\n                Test {\n                    location: \"/test/file.md:34\".to_string(),\n                    header: \"`Test Parsing`::`no output`\".to_string(),\n                    text: \"select * from baz\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: true,\n                    precision_limits: HashMap::new(),\n                },\n                Test {\n                    location: \"/test/file.md:39\".to_string(),\n                    header: \"`Test Parsing`::`end by header`\".to_string(),\n                    text: \"select * from quz\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                },\n                Test {\n                    location: \"/test/file.md:44\".to_string(),\n                    header: \"`Test Parsing`::`end by file`\".to_string(),\n                    text: \"select * from qat\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                },\n            ],\n        };\n        assert!(\n            tests == expected,\n            \"left: {:#?}\\n right: {:#?}\",\n            tests,\n            expected\n        );\n    }\n}\n"
  },
  {
    "path": "tools/sql-doctester/src/runner.rs",
    "content": "use rayon::{iter::ParallelIterator, prelude::*};\n\nuse std::{borrow::Cow, error::Error, fmt};\n\nuse colored::Colorize;\n\nuse postgres::{error::DbError, Client, NoTls, SimpleQueryMessage};\nuse uuid::Uuid;\n\nuse crate::{Test, TestFile};\n\n#[derive(Copy, Clone)]\npub struct ConnectionConfig<'s> {\n    pub host: Option<&'s str>,\n    pub port: Option<&'s str>,\n    pub user: Option<&'s str>,\n    pub password: Option<&'s str>,\n    pub database: Option<&'s str>,\n}\n\nimpl<'s> ConnectionConfig<'s> {\n    fn config_string(&self) -> Cow<'s, str> {\n        use std::fmt::Write;\n\n        let ConnectionConfig {\n            host,\n            port,\n            user,\n            password,\n            database,\n        } = self;\n        let mut config = String::new();\n        if let Some(host) = host {\n            let _ = write!(&mut config, \"host={host} \");\n        }\n        if let Some(port) = port {\n            let _ = write!(&mut config, \"port={port} \");\n        }\n        let _ = match user {\n            Some(user) => write!(&mut config, \"user={user} \"),\n            None => write!(&mut config, \"user=postgres \"),\n        };\n        if let Some(password) = password {\n            let _ = write!(&mut config, \"password={password} \");\n        }\n        if let Some(database) = database {\n            let _ = write!(&mut config, \"dbname={database} \");\n        }\n        Cow::Owned(config)\n    }\n}\n\npub fn run_tests<OnErr: FnMut(Test, TestError)>(\n    connection_config: ConnectionConfig<'_>,\n    startup_script: &str,\n    all_tests: Vec<TestFile>,\n    mut on_error: OnErr,\n) {\n    let root_connection_config = connection_config.config_string();\n    let root_connection_config = &*root_connection_config;\n    eprintln!(\"running {} test files\", all_tests.len());\n\n    let start_db = |tests_name: &str| {\n        let db_name = format!(\"doctest_db__{}\", Uuid::new_v4());\n        let finish_name = tests_name.to_string();\n        let drop_name = db_name.to_string();\n        let deferred = Deferred(move || {\n            eprintln!(\"{} {}\", \"Finished\".bold().green(), finish_name);\n            let _ = Client::connect(root_connection_config, NoTls)\n                .and_then(|mut client| {\n                    client.simple_query(&format!(r#\"DROP DATABASE IF EXISTS \"{drop_name}\"\"#))\n                })\n                .map_err(|e| eprintln!(\"error dropping DB {e}\"));\n        });\n        {\n            eprintln!(\"{} {}\", \"Starting\".bold().green(), tests_name);\n            let mut root_client = Client::connect(root_connection_config, NoTls)\n                .expect(\"could not connect to postgres\");\n            root_client\n                .simple_query(&format!(r#\"CREATE DATABASE \"{db_name}\"\"#))\n                .expect(\"could not create test DB\");\n        }\n        (db_name, deferred)\n    };\n\n    let (stateless_db, _dropper) = match all_tests.iter().any(|t| t.stateless) {\n        false => (None, None),\n        true => {\n            let (name, dropper) = start_db(\"stateless tests\");\n            (Some(name), Some(dropper))\n        }\n    };\n\n    if let Some(db) = stateless_db.as_ref() {\n        let stateless_connection_config = ConnectionConfig {\n            database: Some(db),\n            ..connection_config\n        };\n        let mut client = Client::connect(&stateless_connection_config.config_string(), NoTls)\n            .expect(\"could not connect to test DB\");\n        let _ = client\n            .simple_query(startup_script)\n            .expect(\"could not run init script\");\n    }\n\n    let stateless_db = stateless_db.as_ref();\n\n    let errors: Vec<_> = all_tests\n        .into_par_iter()\n        .flat_map_iter(|tests| {\n            let (db_name, deferred) = match tests.stateless {\n                true => {\n                    eprintln!(\"{} {}\", \"Running\".bold().green(), tests.name);\n                    (stateless_db.map(|s| Cow::Borrowed(&**s)), None)\n                }\n                false => {\n                    let (db_name, deferred) = start_db(&tests.name);\n                    (Some(Cow::Owned(db_name)), Some(deferred))\n                }\n            };\n\n            let test_connection_config = ConnectionConfig {\n                database: db_name.as_deref(),\n                ..connection_config\n            };\n            let mut client = Client::connect(&test_connection_config.config_string(), NoTls)\n                .expect(\"could not connect to test DB\");\n\n            if !tests.stateless {\n                let _ = client\n                    .simple_query(startup_script)\n                    .expect(\"could not run init script\");\n            }\n\n            let deferred = deferred;\n\n            tests.tests.into_iter().map(move |test| {\n                let output = if test.transactional {\n                    run_transactional_test(&mut client, &test)\n                } else {\n                    run_nontransactional_test(&mut client, &test)\n                };\n                // ensure that the DB is dropped after the client\n                let _deferred = &deferred;\n                (test, output)\n            })\n        })\n        .collect();\n\n    drop(_dropper);\n\n    for (test, error) in errors {\n        match error {\n            Ok(..) => continue,\n            Err(error) => on_error(test, error),\n        }\n    }\n}\n\nfn run_transactional_test(client: &mut Client, test: &Test) -> Result<(), TestError> {\n    let mut txn = client.transaction()?;\n    let output = txn.simple_query(&test.text)?;\n    let res = validate_output(output, test);\n    txn.rollback()?;\n    res\n}\n\nfn run_nontransactional_test(client: &mut Client, test: &Test) -> Result<(), TestError> {\n    let output = client.simple_query(&test.text)?;\n    validate_output(output, test)\n}\n\nfn validate_output(output: Vec<SimpleQueryMessage>, test: &Test) -> Result<(), TestError> {\n    use SimpleQueryMessage::*;\n    if test.ignore_output {\n        return Ok(());\n    }\n\n    let mut rows = Vec::with_capacity(test.output.len());\n    for r in output {\n        match r {\n            RowDescription(_r) => continue,\n            Row(r) => {\n                let mut row: Vec<String> = Vec::with_capacity(r.len());\n                for i in 0..r.len() {\n                    row.push(r.get(i).unwrap_or(\"\").to_string())\n                }\n                rows.push(row);\n            }\n            CommandComplete(..) => break,\n            _ => {\n                eprintln!(\"unhandled message: {r:?} for test: {test:?}\");\n                unreachable!()\n            }\n        }\n    }\n    let output_error = |header: &str| {\n        format!(\n            \"{}\\n{expected}\\n{}{}\\n\\n{received}\\n{}{}\\n\\n{delta}\\n{}\",\n            header,\n            stringify_table(&test.output),\n            format!(\"({} rows)\", test.output.len()).dimmed(),\n            stringify_table(&rows),\n            format!(\"({} rows)\", rows.len()).dimmed(),\n            stringify_delta(&test.output, &rows),\n            expected = \"Expected\".bold().blue(),\n            received = \"Received\".bold().blue(),\n            delta = \"Delta\".bold().blue(),\n        )\n    };\n    if test.output.len() != rows.len() {\n        return Err(TestError::OutputError(output_error(\n            \"output has a different number of rows than expected.\",\n        )));\n    }\n\n    fn clamp_len<'s>(mut col: &'s str, idx: usize, test: &Test) -> &'s str {\n        let max_len = test.precision_limits.get(&idx);\n        if let Some(&max_len) = max_len {\n            if col.len() > max_len {\n                col = &col[..max_len]\n            }\n        }\n        col\n    }\n\n    let all_eq = test.output.iter().zip(rows.iter()).all(|(out, row)| {\n        out.len() == row.len()\n            && out\n                .iter()\n                .zip(row.iter())\n                .enumerate()\n                .all(|(i, (o, r))| clamp_len(o, i, test) == clamp_len(r, i, test))\n    });\n    if !all_eq {\n        return Err(TestError::OutputError(output_error(\n            \"output has a different values than expected.\",\n        )));\n    }\n    Ok(())\n}\n\nfn stringify_table(table: &[Vec<String>]) -> String {\n    use std::{cmp::max, fmt::Write};\n    if table.is_empty() {\n        return \"---\".to_string();\n    }\n    let mut width = vec![0; table[0].len()];\n    for row in table {\n        // Ensure that we have width for every column\n        // TODO this shouldn't be needed, but sometimes is?\n        if width.len() < row.len() {\n            width.extend((0..row.len() - width.len()).map(|_| 0));\n        }\n        for (i, value) in row.iter().enumerate() {\n            width[i] = max(width[i], value.len())\n        }\n    }\n    let mut output = String::with_capacity(width.iter().sum::<usize>() + width.len() * 3);\n    for row in table {\n        for (i, value) in row.iter().enumerate() {\n            if i != 0 {\n                output.push_str(\" | \")\n            }\n            let _ = write!(&mut output, \"{:>width$}\", value, width = width[i]);\n        }\n        output.push('\\n')\n    }\n\n    output\n}\n\n#[allow(clippy::needless_range_loop)]\nfn stringify_delta(left: &[Vec<String>], right: &[Vec<String>]) -> String {\n    use std::{cmp::max, fmt::Write};\n\n    static EMPTY_ROW: Vec<String> = vec![];\n    static EMPTY_VAL: String = String::new();\n\n    let mut width = vec![\n        0;\n        max(\n            left.first().map(Vec::len).unwrap_or(0),\n            right.first().map(Vec::len).unwrap_or(0)\n        )\n    ];\n    let num_rows = max(left.len(), right.len());\n    for i in 0..num_rows {\n        let left = left.get(i).unwrap_or(&EMPTY_ROW);\n        let right = right.get(i).unwrap_or(&EMPTY_ROW);\n        let cols = max(left.len(), right.len());\n        for j in 0..cols {\n            let left = left.get(j).unwrap_or(&EMPTY_VAL);\n            let right = right.get(j).unwrap_or(&EMPTY_VAL);\n            if left == right {\n                width[j] = max(width[j], left.len())\n            } else {\n                width[j] = max(width[j], left.len() + right.len() + 2)\n            }\n        }\n    }\n    let mut output = String::with_capacity(width.iter().sum::<usize>() + width.len() * 3);\n    for i in 0..num_rows {\n        let left = left.get(i).unwrap_or(&EMPTY_ROW);\n        let right = right.get(i).unwrap_or(&EMPTY_ROW);\n        let cols = max(left.len(), right.len());\n        for j in 0..cols {\n            let left = left.get(j).unwrap_or(&EMPTY_VAL);\n            let right = right.get(j).unwrap_or(&EMPTY_VAL);\n            if j != 0 {\n                let _ = write!(&mut output, \" | \");\n            }\n            let (value, padding) = if left == right {\n                (left.to_string(), width[j] - left.len())\n            } else {\n                let padding = width[j] - (left.len() + right.len() + 2);\n                let value = format!(\n                    \"{}{}{}{}\",\n                    \"-\".magenta(),\n                    left.magenta(),\n                    \"+\".yellow(),\n                    right.yellow()\n                );\n                (value, padding)\n            };\n            // trick to ensure correct padding, the color characters are counted\n            // if done the normal way.\n            let _ = write!(&mut output, \"{:>padding$}{}\", \"\", value, padding = padding);\n        }\n        let _ = writeln!(&mut output);\n    }\n    output\n}\n\npub enum TestError {\n    PgError(postgres::Error),\n    OutputError(String),\n}\n\nimpl fmt::Display for TestError {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        match self {\n            TestError::PgError(error) => {\n                match error.source().and_then(|e| e.downcast_ref::<DbError>()) {\n                    Some(e) => {\n                        use postgres::error::ErrorPosition::*;\n                        let pos = match e.position() {\n                            Some(Original(pos)) => format!(\"At character {pos}\"),\n                            Some(Internal { position, query }) => {\n                                format!(\"In internal query `{query}` at {position}\")\n                            }\n                            None => String::new(),\n                        };\n                        write!(\n                            f,\n                            \"{}\\n{}\\n{}\\n{}\",\n                            \"Postgres Error:\".bold().red(),\n                            e,\n                            e.detail().unwrap_or(\"\"),\n                            pos,\n                        )\n                    }\n                    None => write!(f, \"{error}\"),\n                }\n            }\n            TestError::OutputError(err) => write!(f, \"{} {err}\", \"Error:\".bold().red()),\n        }\n    }\n}\n\nimpl From<postgres::Error> for TestError {\n    fn from(error: postgres::Error) -> Self {\n        TestError::PgError(error)\n    }\n}\n\nimpl TestError {\n    pub fn annotate_position<'s>(&self, sql: &'s str) -> Cow<'s, str> {\n        match self.location() {\n            None => sql.into(),\n            Some(pos) => format!(\n                \"{}{}{}\",\n                &sql[..pos as usize],\n                \"~>\".bright_red(),\n                &sql[pos as usize..],\n            )\n            .into(),\n        }\n    }\n\n    fn location(&self) -> Option<u32> {\n        use postgres::error::ErrorPosition::*;\n        match self {\n            TestError::OutputError(..) => None,\n            TestError::PgError(e) => match e\n                .source()\n                .and_then(|e| e.downcast_ref::<DbError>().and_then(DbError::position))\n            {\n                None => None,\n                Some(Internal { .. }) => None,\n                Some(Original(pos)) => Some(pos.saturating_sub(1)),\n            },\n        }\n    }\n}\n\nstruct Deferred<T: FnMut()>(T);\n\nimpl<T: FnMut()> Drop for Deferred<T> {\n    fn drop(&mut self) {\n        self.0()\n    }\n}\n"
  },
  {
    "path": "tools/sql-doctester/src/startup.sql",
    "content": "CREATE EXTENSION timescaledb;\nCREATE EXTENSION timescaledb_toolkit;\nSET SESSION TIMEZONE TO 'UTC';\n\n-- utility for generating random numbers\nCREATE SEQUENCE rand START 567;\nCREATE FUNCTION test_random() RETURNS float AS\n    'SELECT ((nextval(''rand'')*34567)%1000)::float/1000'\nLANGUAGE SQL;\n"
  },
  {
    "path": "tools/testbin",
    "content": "#!/bin/sh\n\n# This script automates binary upgrade testing.\n\n# Sample run:\n# OS_NAME=ubuntu OS_VERSION=24.04 tools/testbin -version 1.11.0 -bindir .. -pgversions '13 14' deb\n\n# A released toolkit lists the versions it is upgradeable from in\n# extension/timescaledb_toolkit.control .  This script processes those entries\n# and for each version from which upgrading is supported:\n# 1. Install old binaries (deb or rpm) for each supported postgresql release\n# 2. Run the 1st half of the upgrade tests\n# 3. Install the binary for the version under test\n# 4. Run the 2nd half of the upgrade tests\n\n# The distinction between environment variables and command-line options is\n# possibly inconsistent.  The approach now is for general parameters to come\n# from the command line, and system-specific parameters to come from the\n# environment.  Specifically, these are required in the environment for deb\n# packages only:\n# - OS_NAME\n# - OS_VERSION\n\nset -ex\n\n# Minimum version we support arm64 deb - could possibly go lower.\n# I know 1.8 at least builds on arm.\nMIN_DEB_ARM=1.10.1\n# We added 1: epoch at 1.7.0.\nMIN_DEB_EPOCH=1.7.0\n# TODO Unfortunate that pgrx allows neither specifying nor querying the port it\n#   starts postgres on, so we duplicate that knowledge.  Watch out for\n#   that changing!\nPGRX_PORT_BASE=28800\nCONTROL=extension/timescaledb_toolkit.control\n\n# For PG_VERSIONS default.\n. tools/dependencies.sh\n\nprint() {\n    printf '%s\\n' \"$*\"\n}\n\ndie() {\n    st=${?:-0}\n    if [ $st -eq 0 ]; then\n        st=2\n    fi\n    print \"$*\" >&2\n    exit $st\n}\n\nusage() {\n    die 'testbin [-n] -bindir DIR -version VERSION -pgversions \"[V1] [V2]...\" ( ci | deb | rpm )'\n}\n\n# Requires:\n# - PGRX_PORT_BASE\n# - PG_VERSION\n# Sets:\n# - PG_PORT\nselect_pg() {\n    PG_PORT=$(( $PGRX_PORT_BASE + $PG_VERSION ))\n}\n\n# Start postgres and run the first half (old toolkit) of the test.\n# Must select_pg first.\nstart_test() {\n    $nop cargo pgrx start --package timescaledb_toolkit pg$PG_VERSION\n    $nop cargo run --manifest-path tools/update-tester/Cargo.toml -- create-test-objects -u $LOGNAME -h 127.1 -p $PG_PORT\n}\n\n# Run the second half (new toolkit) of the test and stop postgres.\n# Must select_pg first.\nfinish_test() {\n    $nop cargo run --manifest-path tools/update-tester/Cargo.toml -- validate-test-objects -u $LOGNAME -h 127.1 -p $PG_PORT\n    $nop cargo pgrx stop --package timescaledb_toolkit pg$PG_VERSION\n}\n\ndeb_init() {\n    [ -n \"$OS_NAME\" ] || die 'OS_NAME environment variable must be set to the distribution name e.g. debian or ubuntu'\n    [ -n \"$OS_VERSION\" ] || die 'OS_VERSION environment variable must be set to the distribution version number'\n\n    ARCH=`dpkg --print-architecture`\n    EPOCH=\n    MIN_DEB_ARM=`cmp_version $MIN_DEB_ARM`\n    MIN_DEB_EPOCH=`cmp_version $MIN_DEB_EPOCH`\n}\n\n# Requires:\n# - FROM_VERSION\nskip_from_version() {\n    # We released 1.10.0-dev by accident.  We have to support upgrades\n    # from it (and we tested that at the time), but we pulled the binaries, so\n    # we can't test it here.\n    [ $FROM_VERSION = 1.10.0-dev ] && return\n    [ $OS_NAME = debian ] && [ $OS_VERSION = 10 ] && [ `cmp_version $FROM_VERSION` -lt 011100 ] && return\n    [ $OS_NAME = ubuntu ] && [ $OS_VERSION = 22.04 ] && [ `cmp_version $FROM_VERSION` -lt 010600 ] && return\n}\n\n# Requires:\n# - FROM_VERSION\n# - PG_VERSION\nskip_from_version_pg_version() {\n    # skip versions without binaries for this PostgreSQL version\n    [ $PG_VERSION -gt 14 ] && [ `cmp_version $FROM_VERSION` -lt 011301 ] && return\n    [ $PG_VERSION -gt 15 ] && [ `cmp_version $FROM_VERSION` -lt 011801 ] && return\n}\n\n# Requires:\n# - FROM_VERSION\ndeb_start_test() {\n    skip_from_version && return 1\n    cmp_version=`cmp_version $FROM_VERSION`\n    [ \"$ARCH\" = arm64 ] && [ $cmp_version -lt $MIN_DEB_ARM ] && return 1\n\n    [ $cmp_version -ge $MIN_DEB_EPOCH ] && EPOCH=1:\n    for PG_VERSION in $PG_VERSIONS; do\n        skip_from_version_pg_version && continue\n        select_pg $PG_VERSION\n        deb=timescaledb-toolkit-postgresql-${PG_VERSION}=${EPOCH}${FROM_VERSION}~${OS_NAME}${OS_VERSION}\n        $nop sudo apt-get -qq install $deb || die\n\n        start_test || die\n    done\n}\n\ntest_deb() {\n    deb_init\n    for FROM_VERSION; do\n        deb_start_test || continue\n        for PG_VERSION in $PG_VERSIONS; do\n            skip_from_version_pg_version && continue\n            select_pg $PG_VERSION\n            deb=timescaledb-toolkit-postgresql-${PG_VERSION}_${TOOLKIT_VERSION}~${OS_NAME}${OS_VERSION}_${ARCH}.deb\n            $nop sudo dpkg -i \"$BINDIR/$deb\"\n\n            finish_test\n\n            $nop sudo dpkg -P timescaledb-toolkit-postgresql-$PG_VERSION\n        done\n    done\n}\n\ntest_ci() {\n    deb_init\n\n    # When run under CI after a recent release, the Packages file in the\n    # container image don't know about the latest version.\n    $nop sudo apt-get update\n\n    for FROM_VERSION; do\n        deb_start_test || continue\n        for PG_VERSION in $PG_VERSIONS; do\n            skip_from_version_pg_version && continue\n            select_pg $PG_VERSION\n            $nop sudo dpkg -P timescaledb-toolkit-postgresql-$PG_VERSION\n            # Installing (and possibly uninstalling) toolkit binary gives this back to root but we need to write to it.\n            $nop sudo chown $LOGNAME /usr/lib/postgresql/$PG_VERSION/lib /usr/share/postgresql/$PG_VERSION/extension\n            $nop tools/build -pg$PG_VERSION install\n\n            finish_test\n        done\n    done\n}\n\nrpm_start_test() {\n    for PG_VERSION in $PG_VERSIONS; do\n        skip_from_version_pg_version && continue\n        select_pg $PG_VERSION\n        rpm=timescaledb-toolkit-postgresql-$PG_VERSION\n        # yum doesn't seem to allow force-install of a specific version.\n        # If the package is already installed at a different version,\n        # the install command below does nothing.\n        # So, uninstall if installed.\n        $nop rpm -q $rpm > /dev/null && $nop sudo rpm -e $rpm\n        $nop sudo yum -q -y install $rpm-$FROM_VERSION\n\n        start_test\n    done\n}\n\ntest_rpm() {\n    ARCH=`rpm -E '%{_arch}'`\n    for FROM_VERSION; do\n        skip_from_version && continue\n        rpm_start_test\n        for PG_VERSION in $PG_VERSIONS; do\n            skip_from_version_pg_version && continue\n            select_pg $PG_VERSION\n            rpm=timescaledb-toolkit-postgresql-$PG_VERSION-$TOOLKIT_VERSION-0.el$OS_VERSION.$ARCH.rpm\n            $nop sudo rpm -U \"$BINDIR/$rpm\"\n\n            finish_test\n\n            $nop sudo rpm -e timescaledb-toolkit-postgresql-$PG_VERSION\n        done\n    done\n}\n\ntest_rpm_ci() {\n    for FROM_VERSION; do\n        skip_from_version && continue\n        rpm_start_test\n        for PG_VERSION in $PG_VERSIONS; do\n            skip_from_version_pg_version && continue\n            select_pg $PG_VERSION\n\n            $nop sudo rpm -e timescaledb-toolkit-postgresql-$PG_VERSION\n            $nop sudo chown -R $LOGNAME /usr/pgsql-$PG_VERSION/lib /usr/pgsql-$PG_VERSION/share/extension\n            $nop tools/build -pg$PG_VERSION install\n\n            finish_test\n        done\n    done\n}\n\n# Format 3-part version string for numeric comparison.\n# If this script has survived to see one of the 3 parts incremented past 99:\n# congratulations!  It is not hard to fix.\ncmp_version() {\n    minpat=${1#*.}\n    printf '%02d%02d%02d' ${1%%.*} ${minpat%.*} ${minpat#*.} 2> /dev/null\n}\n\nprint_upgradeable_from() {\n    # TODO We never shipped a 1.4 deb and the 1.5 deb is called 1.5.0\n    #  Let's draw the line there and remove those from upgradeable_from.\n    #  Someone who needs to upgrade from 1.4 or 1.5 can upgrade to 1.10.1 and then beyond.\n    sed -n \"s/'//g; s/,//g; s/^# upgradeable_from = 1\\.4 1\\.5 //p\" $CONTROL\n}\n\ncleanup() {\n    set +e\n    for PG_VERSION in $PG_VERSIONS; do\n        select_pg $PG_VERSION\n        $nop cargo pgrx stop --package timescaledb_toolkit pg$PG_VERSION\n    done\n}\n\nrun() {\n    [ -n \"$LOGNAME\" ] || die 'LOGNAME environment variable must be set to the login name'\n    [ -n \"$PG_VERSIONS\" ] || die '-pgversions required'\n    # TODO Requiring -bindir and -version when not all methods need them is awkward but eh.\n    [ -d \"$BINDIR\" ] || die '-bindir required'\n    [ -n \"$TOOLKIT_VERSION\" ] || die '-version required'\n\n    trap cleanup 0\n    test_$1 `print_upgradeable_from`\n    trap - 0\n\n    echo DONE\n}\n\nwhile [ $# -gt 0 ]; do\n    arg=\"$1\"\n    shift\n    case \"$arg\" in\n        -n)\n            nop=:\n            ;;\n\n        -bindir)\n            BINDIR=$1\n            shift\n            ;;\n\n        -pgversions)\n            PG_VERSIONS=$1\n            shift\n            ;;\n\n        -version)\n            TOOLKIT_VERSION=$1\n            shift\n            ;;\n\n        ci|deb|rpm|rpm_ci)\n            run $arg\n            ;;\n\n        *)\n            usage\n            ;;\n    esac\ndone\n"
  },
  {
    "path": "tools/update-tester/Cargo.toml",
    "content": "[package]\nname = \"update-tester\"\nversion = \"0.3.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\ncontrol_file_reader = {path = \"../../crates/scripting-utilities/control_file_reader\"}\npostgres_connection_configuration = {path = \"../../crates/scripting-utilities/postgres_connection_configuration\"}\n\ncolored = \"2.0.0\"\nclap = { version = \"3.2.15\", features = [\"wrap_help\"] }\npostgres = \"0.19.1\"\nsemver = \"1.0.9\"\ntoml_edit = \"0.14.3\"\nxshell = \"0.1.17\"\npulldown-cmark = \"0.8.0\"\nwalkdir = \"2.3.2\"\nbytecount = \"0.6.3\"\n"
  },
  {
    "path": "tools/update-tester/Readme.md",
    "content": "# Update Tester #\n\nRuns update tests. It'll install every version of the extension marked as\n`upgradeable_from` in `timescaledb_toolkit.control` and test that updates to\nthe current version work correctly. At a high level:\n\n1. For each version in `upgradeable_from`\n    1. Checkout the corresponding tag in git,\n    2. Build and install the extension at that tag,\n    3. Set git back to the original state.\n2. Build and install the extension at the original git state.\n3. For each version in `upgradeable_from`\n    1. create a database,\n    2. install the old version of the extension,\n    3. install some `timescaledb_toolkit` objects,\n    4. update the extension,\n    5. validate the extension is in the expected state.\n\n**NOTE:** Running this _will_ move git's `HEAD`. Though git will warn on\n          conflicts, and we do our best to reset the tree state before the\n          script exits, we recommend only using it on a clean tree.\n\n\n\n\n```\nUSAGE:\n    update-tester [OPTIONS] <dir> <pg_config> <cargo_pgrx> <cargo_pgrx_old>\n\nFLAGS:\n        --help       Prints help information\n    -V, --version    Prints version information\n\nOPTIONS:\n    -d, --database <database>    postgres database the root connection should use. By\n                                 default this DB will only be used to spawn the individual\n                                 test databases; no tests will run against it.\n    -h, --host <hostname>        postgres host\n    -a, --password <password>    postgres password\n    -p, --port <portnumber>      postgres port\n    -u, --user <username>        postgres user\n\nARGS:\n    <dir>              Path in which to find the timescaledb-toolkit repo\n    <pg_config>        Path to pg_config for the DB we are using\n    <cargo_pgrx>        Path to cargo-pgrx (must be 0.4 series or newer)\n    <cargo_pgrx_old>    Path to cargo-pgrx 0.2-0.3 series\n```\n"
  },
  {
    "path": "tools/update-tester/src/installer.rs",
    "content": "#![allow(unexpected_cfgs)]\n\nuse std::{collections::HashSet, path::Path};\n\nuse colored::Colorize;\nuse semver::Version;\nuse toml_edit::Document;\nuse xshell::{cmd, cp, mkdir_p, pushd, read_dir};\n\nuse crate::{defer, quietly_run};\n\nfn pgrx_name(version: &Version) -> &'static str {\n    if version >= &Version::new(0, 7, 4) {\n        \"pgx\"\n    } else {\n        \"pgrx\"\n    }\n}\n\n#[allow(clippy::too_many_arguments)]\npub fn install_all_versions(\n    root_dir: &str,\n    cache_dir: Option<&str>,\n    pg_config: &str,\n    cargo_pgrx: &str,\n    cargo_pgrx_old: &str,\n    current_version: &str,\n    old_versions: &[String],\n    reinstall: &HashSet<&str>,\n) -> xshell::Result<()> {\n    let extension_dir = path!(root_dir / \"extension\");\n    let install_toolkit = |pgrx_version: Version| -> xshell::Result<()> {\n        let _d = pushd(&extension_dir)?;\n        let pgrx_name = pgrx_name(&pgrx_version);\n        match pgrx_version >= Version::new(0, 4, 0) {\n            true => quietly_run(cmd!(\"{cargo_pgrx} {pgrx_name} install -c {pg_config}\")),\n            false => quietly_run(cmd!(\"{cargo_pgrx_old} {pgrx_name} install -c {pg_config}\")),\n        }\n    };\n    let post_install = || -> xshell::Result<()> {\n        let _d = pushd(root_dir)?;\n        quietly_run(cmd!(\n            \"cargo run --manifest-path ./tools/post-install/Cargo.toml -- {pg_config}\"\n        ))\n    };\n\n    if let Some(cache_dir) = cache_dir {\n        restore_from_cache(cache_dir, pg_config)?\n    }\n\n    {\n        let base_checkout = get_current_checkout()?;\n        let pgrx_version = get_pgrx_version(\n            &std::fs::read_to_string(\"extension/Cargo.toml\").expect(\"unable to read Cargo.toml\"),\n        );\n        // Install the versions in reverse-time order.\n        // Since later versions tend to be supersets of old versions,\n        // I expect compilation to be faster this way - Josh\n        for version in old_versions.iter().rev() {\n            let force_reinstall = reinstall.contains(&**version);\n            if !force_reinstall && version_is_installed(pg_config, version)? {\n                eprintln!(\"{} {}\", \"Already Installed\".blue(), version);\n                continue;\n            }\n            eprintln!(\"{} {}\", \"Installing\".bold().cyan(), version);\n            let tag_version = tag_version(version);\n            quietly_run(cmd!(\"git fetch origin tag {tag_version}\"))?;\n            quietly_run(cmd!(\"git checkout tags/{tag_version}\"))?;\n            let _d = defer(|| quietly_run(cmd!(\"git checkout {base_checkout}\")));\n            let pgrx_version = get_pgrx_version(\n                &std::fs::read_to_string(\"extension/Cargo.toml\")\n                    .expect(\"unable to read Cargo.toml\"),\n            );\n            install_toolkit(pgrx_version)?;\n            post_install()?;\n            eprintln!(\"{} {}\", \"Finished\".bold().green(), version);\n        }\n\n        if let Some(cache_dir) = cache_dir {\n            save_to_cache(cache_dir, pg_config)?;\n        }\n\n        eprintln!(\n            \"{} {} ({})\",\n            \"Installing Current\".bold().cyan(),\n            current_version,\n            base_checkout\n        );\n        install_toolkit(pgrx_version)?;\n    }\n    post_install()?;\n    eprintln!(\"{}\", \"Finished Current\".bold().green());\n\n    Ok(())\n}\n\nfn get_current_checkout() -> xshell::Result<String> {\n    let current_branch = cmd!(\"git rev-parse --abbrev-ref --symbolic-full-name HEAD\").read()?;\n\n    if current_branch != \"HEAD\" {\n        return Ok(current_branch);\n    }\n\n    cmd!(\"git rev-parse --verify HEAD\").read()\n}\n\nfn get_pgrx_version(cargo_toml_contents: &str) -> Version {\n    let cargo = cargo_toml_contents\n        .parse::<Document>()\n        .expect(\"invalid Cargo.toml\");\n\n    let pgrx_dependency = cargo[\"dependencies\"]\n        .get(\"pgrx\")\n        // check old name if no pgrx found\n        .unwrap_or_else(|| &cargo[\"dependencies\"][\"pgx\"]);\n    pgrx_dependency\n        .as_str()\n        .expect(\"expected pgrx to only have a version\")\n        .trim_start_matches(['=', '^', '~'].as_slice())\n        .parse()\n        .expect(\"cannot parse pgrx version\")\n}\n\n// We were unprincipled with some of our old versions, so the version from\n// the control file is `x.y`, while the tag is `x.y.0`. This function translates\n// from the control file version to the tag version (in a rather hacky way)\nfn tag_version(version: &str) -> String {\n    if version.matches('.').count() >= 2 {\n        return version.into();\n    }\n\n    format!(\"{version}.0\")\n}\n\n//-----------------------//\n//-- Cache Maintenance --//\n//-----------------------//\n\nfn version_is_installed(pg_config: &str, version: &str) -> xshell::Result<bool> {\n    let binary_name = format!(\"timescaledb_toolkit-{version}.so\");\n    let bin_dir = cmd!(\"{pg_config} --pkglibdir\").read()?;\n    let installed_files = read_dir(bin_dir)?;\n    let installed = installed_files.into_iter().any(|file| {\n        file.file_name()\n            .map(|name| name.to_string_lossy() == binary_name)\n            .unwrap_or(false)\n    });\n    Ok(installed)\n}\n\nfn restore_from_cache(cache_dir: &str, pg_config: &str) -> xshell::Result<()> {\n    if !path!(cache_dir).exists() {\n        eprintln!(\"{}\", \"Cache does not exist\".yellow());\n        return Ok(());\n    }\n\n    eprintln!(\"{} {}\", \"Restoring from Cache\".bold().blue(), cache_dir);\n    let bin_dir = cmd!(\"{pg_config} --pkglibdir\").read()?;\n\n    let share_dir = cmd!(\"{pg_config} --sharedir\").read()?;\n    let script_dir = path!(share_dir / \"extension\");\n\n    let cached_bin_dir = path!(cache_dir / \"bin\");\n    let cached_script_dir = path!(cache_dir / \"extension\");\n\n    cp_dir(cached_bin_dir, bin_dir, |_| true)?;\n    cp_dir(cached_script_dir, script_dir, |_| true)\n}\n\nfn save_to_cache(cache_dir: &str, pg_config: &str) -> xshell::Result<()> {\n    eprintln!(\"{} {}\", \"Saving to Cache\".blue(), cache_dir);\n\n    let cached_bin_dir = path!(cache_dir / \"bin\");\n    let cached_script_dir = path!(cache_dir / \"extension\");\n\n    if !cached_bin_dir.exists() {\n        mkdir_p(&cached_bin_dir)?\n    }\n\n    if !cached_script_dir.exists() {\n        mkdir_p(&cached_script_dir)?\n    }\n\n    let bin_dir = cmd!(\"{pg_config} --pkglibdir\").read()?;\n\n    let share_dir = cmd!(\"{pg_config} --sharedir\").read()?;\n    let script_dir = path!(share_dir / \"extension\");\n\n    let is_toolkit_file = |file: &Path| {\n        file.file_name()\n            .map(|f| f.to_string_lossy().starts_with(\"timescaledb_toolkit\"))\n            .unwrap_or(false)\n    };\n    cp_dir(bin_dir, cached_bin_dir, is_toolkit_file)?;\n    cp_dir(script_dir, cached_script_dir, is_toolkit_file)\n}\n\nfn cp_dir(\n    src: impl AsRef<Path>,\n    dst: impl AsRef<Path>,\n    mut filter: impl FnMut(&Path) -> bool,\n) -> xshell::Result<()> {\n    let dst = dst.as_ref();\n    for file in read_dir(src)? {\n        if filter(&file) {\n            cp(file, dst)?;\n        }\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "tools/update-tester/src/main.rs",
    "content": "use std::{\n    collections::HashSet,\n    io::{self, Write},\n    path::Path,\n    process,\n};\n\nuse clap::Arg;\nuse clap::Command;\n\nuse colored::Colorize;\n\nuse xshell::{read_file, Cmd};\n\nuse control_file_reader::get_upgradeable_from;\nuse postgres_connection_configuration::ConnectionConfig;\n\n// macro for literate path joins\nmacro_rules! path {\n    ($start:ident $(/ $segment: literal)*) => {\n        {\n            let root: &Path = $start.as_ref();\n            root $(.join($segment))*\n        }\n    };\n    ($start:ident / $segment: expr) => {\n        {\n            let root: &Path = $start.as_ref();\n            root.join($segment)\n        }\n    }\n}\n\nmod installer;\nmod parser;\nmod testrunner;\n\nfn main() {\n    let matches = Command::new(\"update-tester\")\n        .about(\"Update tester for toolkit releases\")\n        .subcommand_required(true)\n        .arg_required_else_help(true)\n\t.subcommand(\n            Command::new(\"full-update-test-source\")\n            .long_flag(\"full-update-test-source\")\n            .about(\"Run update-test, building toolkit from source unless a local cache is supplied\")\n            .arg(\n                Arg::new(\"HOST\")\n                    .short('h')\n                    .long(\"host\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"PORT\")\n                    .short('p')\n                    .long(\"port\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"USER\")\n                    .short('u')\n                    .long(\"user\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"PASSWORD\")\n                    .short('a')\n                    .long(\"password\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"DB\")\n                    .short('d')\n                    .long(\"database\")\n                    .takes_value(true)\n            )\n            .arg(Arg::new(\"CACHE\").short('c').long(\"cache\").takes_value(true))\n            .arg(Arg::new(\"REINSTALL\").long(\"reinstall\").takes_value(true))\n            .arg(Arg::new(\"PG_CONFIG\").takes_value(true))\n            .arg(Arg::new(\"CARGO_PGRX\").takes_value(true))\n            .arg(Arg::new(\"CARGO_PGRX_OLD\").takes_value(true)),\n    )\n\t.subcommand(\n\t    Command::new(\"create-test-objects\")\n            .long_flag(\"create-test-objects\")\n            .about(\"Creates test objects in a db using the currently installed version of Toolkit\")\n            .arg(\n                Arg::new(\"HOST\")\n                    .short('h')\n                    .long(\"host\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"PORT\")\n                    .short('p')\n                    .long(\"port\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"USER\")\n                    .short('u')\n                    .long(\"user\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"PASSWORD\")\n                    .short('a')\n                    .long(\"password\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"DB\")\n                    .short('d')\n                    .long(\"database\")\n                    .takes_value(true)\n            )\n\t)\n\t.subcommand(\n\t    Command::new(\"validate-test-objects\")\n            .long_flag(\"validate-test-objects\")\n            .about(\"Runs a series of checks on the objects created by create-test-objects using the currently installed version of Toolkit\")\n            .arg(\n                Arg::new(\"HOST\")\n                    .short('h')\n                    .long(\"host\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"PORT\")\n                    .short('p')\n                    .long(\"port\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"USER\")\n                    .short('u')\n                    .long(\"user\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"PASSWORD\")\n                    .short('a')\n                    .long(\"password\")\n                    .takes_value(true)\n            )\n            .arg(\n                Arg::new(\"DB\")\n                    .short('d')\n                    .long(\"database\")\n                    .takes_value(true)\n            )\n\t)\n// Mutates help, removing the short flag (-h) so that it can be used by HOST\n\t.mut_arg(\"help\", |_h| {\n      Arg::new(\"help\")\n          .long(\"help\")\n  })\n\t.get_matches();\n\n    match matches.subcommand() {\n        Some((\"full-update-test-source\", full_update_matches)) => {\n            let connection_config = ConnectionConfig {\n                host: full_update_matches.value_of(\"HOST\"),\n                port: full_update_matches.value_of(\"PORT\"),\n                user: full_update_matches.value_of(\"USER\"),\n                password: full_update_matches.value_of(\"PASSWORD\"),\n                database: full_update_matches.value_of(\"DB\"),\n            };\n\n            let cache_dir = full_update_matches.value_of(\"CACHE\");\n\n            let root_dir = \".\";\n\n            let reinstall = full_update_matches\n                .value_of(\"REINSTALL\")\n                .map(|r| r.split_terminator(',').collect())\n                .unwrap_or_else(HashSet::new);\n\n            let pg_config = full_update_matches\n                .value_of(\"PG_CONFIG\")\n                .expect(\"missing pg_config\");\n            let cargo_pgrx = full_update_matches\n                .value_of(\"CARGO_PGRX\")\n                .expect(\"missing cargo_pgrx\");\n            let cargo_pgrx_old = full_update_matches\n                .value_of(\"CARGO_PGRX_OLD\")\n                .expect(\"missing cargo_pgrx_old\");\n\n            let mut num_errors = 0;\n            let on_error = |test: parser::Test, error: testrunner::TestError| {\n                num_errors += 1;\n                eprintln!(\n                    \"{} {}\\n\",\n                    test.location.bold().blue(),\n                    test.header.bold().dimmed()\n                );\n                eprintln!(\"{}\", error.annotate_position(&test.text));\n                eprintln!(\"{error}\\n\");\n            };\n\n            let res = try_main(\n                root_dir,\n                cache_dir,\n                &connection_config,\n                pg_config,\n                cargo_pgrx,\n                cargo_pgrx_old,\n                reinstall,\n                on_error,\n            );\n            if let Err(err) = res {\n                eprintln!(\"{err}\");\n                process::exit(1);\n            }\n            if num_errors > 0 {\n                eprintln!(\n                    \"{} {}\\n\",\n                    num_errors.to_string().bold().red(),\n                    \"Tests Failed\".bold().red()\n                );\n                process::exit(1)\n            }\n            eprintln!(\"{}\\n\", \"Tests Passed\".bold().green());\n        }\n        Some((\"create-test-objects\", create_test_object_matches)) => {\n            let connection_config = ConnectionConfig {\n                host: create_test_object_matches.value_of(\"HOST\"),\n                port: create_test_object_matches.value_of(\"PORT\"),\n                user: create_test_object_matches.value_of(\"USER\"),\n                password: create_test_object_matches.value_of(\"PASSWORD\"),\n                database: create_test_object_matches.value_of(\"DB\"),\n            };\n\n            let mut num_errors = 0;\n            let on_error = |test: parser::Test, error: testrunner::TestError| {\n                num_errors += 1;\n                eprintln!(\n                    \"{} {}\\n\",\n                    test.location.bold().blue(),\n                    test.header.bold().dimmed()\n                );\n                eprintln!(\"{}\", error.annotate_position(&test.text));\n                eprintln!(\"{error}\\n\");\n            };\n            let res = try_create_objects(&connection_config, on_error);\n            if let Err(err) = res {\n                eprintln!(\"{err}\");\n                process::exit(1);\n            }\n            if num_errors > 0 {\n                eprintln!(\n                    \"{} {} {}\\n\",\n                    \"Object Creation Failed With\".bold().red(),\n                    num_errors.to_string().bold().red(),\n                    \"Errors\".bold().red()\n                );\n                process::exit(1)\n            }\n            eprintln!(\"{}\\n\", \"Objects Created Successfully\".bold().green());\n        }\n        Some((\"validate-test-objects\", validate_test_object_matches)) => {\n            let connection_config = ConnectionConfig {\n                host: validate_test_object_matches.value_of(\"HOST\"),\n                port: validate_test_object_matches.value_of(\"PORT\"),\n                user: validate_test_object_matches.value_of(\"USER\"),\n                password: validate_test_object_matches.value_of(\"PASSWORD\"),\n                database: validate_test_object_matches.value_of(\"DB\"),\n            };\n\n            let mut num_errors = 0;\n            let on_error = |test: parser::Test, error: testrunner::TestError| {\n                num_errors += 1;\n                eprintln!(\n                    \"{} {}\\n\",\n                    test.location.bold().blue(),\n                    test.header.bold().dimmed()\n                );\n                eprintln!(\"{}\", error.annotate_position(&test.text));\n                eprintln!(\"{error}\\n\");\n            };\n\n            let root_dir = \".\";\n            let res = try_validate_objects(&connection_config, root_dir, on_error);\n            if let Err(err) = res {\n                eprintln!(\"{err}\");\n                process::exit(1);\n            }\n            if num_errors > 0 {\n                eprintln!(\"{num_errors} {}\\n\", \"Tests Failed\".bold().red());\n                eprintln!(\"{}\\n\", \"Validation Failed\".bold().red());\n                process::exit(1)\n            }\n\n            eprintln!(\"{}\\n\", \"Validations Completed Successfully\".bold().green());\n        }\n        _ => unreachable!(), // if all subcommands are defined, anything else is unreachable\n    }\n}\n\n#[allow(clippy::too_many_arguments)]\nfn try_main<OnErr: FnMut(parser::Test, testrunner::TestError)>(\n    root_dir: &str,\n    cache_dir: Option<&str>,\n    db_conn: &ConnectionConfig<'_>,\n    pg_config: &str,\n    cargo_pgrx: &str,\n    cargo_pgrx_old: &str,\n    reinstall: HashSet<&str>,\n    on_error: OnErr,\n) -> xshell::Result<()> {\n    let (current_version, old_versions) = get_version_info(root_dir)?;\n    if old_versions.is_empty() {\n        panic!(\"no old versions to upgrade from\")\n    }\n\n    println!(\"{} [{}]\", \"Testing\".green().bold(), old_versions.join(\", \"));\n\n    installer::install_all_versions(\n        root_dir,\n        cache_dir,\n        pg_config,\n        cargo_pgrx,\n        cargo_pgrx_old,\n        &current_version,\n        &old_versions,\n        &reinstall,\n    )?;\n\n    testrunner::run_update_tests(db_conn, current_version, old_versions, on_error)\n}\nfn try_create_objects<OnErr: FnMut(parser::Test, testrunner::TestError)>(\n    db_conn: &ConnectionConfig<'_>,\n    on_error: OnErr,\n) -> xshell::Result<()> {\n    testrunner::create_test_objects_for_package_testing(db_conn, on_error)\n}\n\nfn try_validate_objects<OnErr: FnMut(parser::Test, testrunner::TestError)>(\n    _conn: &ConnectionConfig<'_>,\n    root_dir: &str,\n    on_error: OnErr,\n) -> xshell::Result<()> {\n    let (_current_version, old_versions) = get_version_info(root_dir)?;\n    if old_versions.is_empty() {\n        panic!(\"no old versions to upgrade from\")\n    }\n    testrunner::update_to_and_validate_new_toolkit_version(_conn, on_error)\n}\n\nfn get_version_info(root_dir: &str) -> xshell::Result<(String, Vec<String>)> {\n    let extension_dir = path!(root_dir / \"extension\");\n    let control_file = path!(extension_dir / \"timescaledb_toolkit.control\");\n    let manifest_file = path!(extension_dir / \"Cargo.toml\");\n\n    let manifest_contents = read_file(manifest_file)?;\n    let control_contents = read_file(control_file)?;\n\n    let current_version = manifest_contents\n        .parse::<toml_edit::Document>()\n        .expect(\"failed to parse extension/Cargo.toml\")\n        .get(\"package\")\n        .expect(\"failed to find [package] in extension/Cargo.toml\")\n        .get(\"version\")\n        .expect(\"failed to find package.version in extension/Cargo.toml\")\n        .as_str()\n        .expect(\"package.version not a string in extension/Cargo.toml\")\n        .to_owned();\n\n    let upgradable_from = get_upgradeable_from(&control_contents)\n        .unwrap_or_else(|e| panic!(\"{e} in control file {control_contents}\"));\n\n    Ok((current_version, upgradable_from))\n}\n\n//-------------//\n//- Utilities -//\n//-------------//\n\n// run a command, only printing the output on failure\nfn quietly_run(cmd: Cmd) -> xshell::Result<()> {\n    let display = format!(\"{cmd}\");\n    let output = cmd.ignore_status().output()?;\n    if !output.status.success() {\n        io::stdout()\n            .write_all(&output.stdout)\n            .expect(\"cannot write to stdout\");\n        io::stdout()\n            .write_all(&output.stderr)\n            .expect(\"cannot write to stdout\");\n        panic!(\n            \"{} `{display}` exited with a non-zero error code {}\",\n            \"ERROR\".bold().red(),\n            output.status\n        )\n    }\n    Ok(())\n}\n\n// run a command on `drop()`\nfn defer<T>(f: impl FnMut() -> T) -> Deferred<T, impl FnMut() -> T> {\n    Deferred(f)\n}\n\nstruct Deferred<T, F: FnMut() -> T>(F);\n\nimpl<F, T> Drop for Deferred<T, F>\nwhere\n    F: FnMut() -> T,\n{\n    fn drop(&mut self) {\n        self.0();\n    }\n}\n"
  },
  {
    "path": "tools/update-tester/src/parser.rs",
    "content": "use std::{collections::HashMap, ffi::OsStr, fs, path::Path};\n\nuse pulldown_cmark::{\n    CodeBlockKind::Fenced,\n    CowStr, Event, Parser,\n    Tag::{CodeBlock, Heading},\n};\nuse semver::Version;\n\n#[derive(Debug, PartialEq, Eq)]\n#[must_use]\npub struct TestFile {\n    pub name: String,\n    stateless: bool,\n    pub tests: Vec<Test>,\n}\n\n#[derive(Debug, PartialEq, Eq, Clone)]\n#[must_use]\npub struct Test {\n    pub location: String,\n    pub header: String,\n    pub text: String,\n    pub output: Vec<Vec<String>>,\n    transactional: bool,\n    ignore_output: bool,\n    pub precision_limits: HashMap<usize, usize>,\n    pub creation: bool,\n    pub validation: bool,\n    pub min_toolkit_version: Option<Version>,\n}\n\npub fn extract_tests(root: &str) -> Vec<TestFile> {\n    // TODO handle when root is a file\n    let mut all_tests = vec![];\n    let walker = walkdir::WalkDir::new(root)\n        .follow_links(true)\n        .sort_by(|a, b| a.path().cmp(b.path()));\n    for entry in walker {\n        let entry = entry.unwrap();\n        if !entry.file_type().is_file() {\n            continue;\n        }\n\n        if entry.path().extension() != Some(OsStr::new(\"md\")) {\n            continue;\n        }\n\n        let contents = fs::read_to_string(entry.path()).unwrap();\n\n        let tests = extract_tests_from_string(&contents, &entry.path().to_string_lossy());\n        if !tests.tests.is_empty() {\n            all_tests.push(tests)\n        }\n    }\n    all_tests\n}\n// parsers the grammar `(heading* (test output?)*)*`\npub fn extract_tests_from_string(s: &str, file_stem: &str) -> TestFile {\n    let mut parser = Parser::new(s).into_offset_iter().peekable();\n    let mut heading_stack: Vec<String> = vec![];\n    let mut tests = vec![];\n\n    let mut last_test_seen_at = 0;\n    let mut lines_seen = 0;\n\n    let mut stateless = true;\n\n    // consume the parser until an tag is reached, performing an action on each text\n    macro_rules! consume_text_until {\n        ($parser: ident yields $end: pat => $action: expr) => {\n            for (event, _) in &mut parser {\n                match event {\n                    Event::Text(text) => $action(text),\n                    $end => break,\n                    _ => (),\n                }\n            }\n        };\n    }\n\n    'block_hunt: while let Some((event, span)) = parser.next() {\n        match event {\n            // we found a heading, add it to the stack\n            Event::Start(Heading(level)) => {\n                heading_stack.truncate(level as usize - 1);\n                let mut header = \"`\".to_string();\n                consume_text_until!(parser yields Event::End(Heading(..)) =>\n                    |text: CowStr| header.push_str(&text)\n                );\n                header.truncate(header.trim_end().len());\n                header.push('`');\n                heading_stack.push(header);\n            }\n\n            // we found a code block, if it's a test add the test\n            Event::Start(CodeBlock(Fenced(ref info))) => {\n                let code_block_info = parse_code_block_info(info);\n\n                // non-test code block, consume it and continue looking\n                if let BlockKind::Other = code_block_info.kind {\n                    for (event, _) in &mut parser {\n                        if let Event::End(CodeBlock(Fenced(..))) = event {\n                            break;\n                        }\n                    }\n                    continue 'block_hunt;\n                }\n\n                let current_line = {\n                    let offset = span.start;\n                    lines_seen += bytecount::count(&s.as_bytes()[last_test_seen_at..offset], b'\\n');\n                    last_test_seen_at = offset;\n                    lines_seen + 1\n                };\n\n                if let BlockKind::Output = code_block_info.kind {\n                    panic!(\n                        \"found output with no test test.\\n{file_stem}:{current_line} {heading_stack:?}\"\n                    )\n                }\n\n                assert!(matches!(code_block_info.kind, BlockKind::Sql));\n\n                stateless &= code_block_info.transactional;\n                let mut test = Test {\n                    location: format!(\"{file_stem}:{current_line}\"),\n                    header: if heading_stack.is_empty() {\n                        \"<root>\".to_string()\n                    } else {\n                        heading_stack.join(\"::\")\n                    },\n                    text: String::new(),\n                    output: Vec::new(),\n                    transactional: code_block_info.transactional,\n                    ignore_output: code_block_info.ignore_output,\n                    precision_limits: code_block_info.precision_limits,\n                    min_toolkit_version: code_block_info.min_toolkit_version,\n                    creation: code_block_info.creation,\n                    validation: code_block_info.validation,\n                };\n\n                // consume the lines of the test\n                consume_text_until!(parser yields Event::End(CodeBlock(Fenced(..))) =>\n                    |text: CowStr| test.text.push_str(&text)\n                );\n\n                // search to see if we have output\n                loop {\n                    match parser.peek() {\n                        // we found a code block, is it output?\n                        Some((Event::Start(CodeBlock(Fenced(info))), _)) => {\n                            let code_block_info = parse_code_block_info(info);\n                            match code_block_info.kind {\n                                // non-output, continue at the top\n                                BlockKind::Sql | BlockKind::Other => {\n                                    tests.push(test);\n                                    continue 'block_hunt;\n                                }\n\n                                // output, consume it\n                                BlockKind::Output => {\n                                    if !test.precision_limits.is_empty()\n                                        && !code_block_info.precision_limits.is_empty()\n                                    {\n                                        panic!(\n                                            \"cannot have precision limits on both test and output.\\n{file_stem}:{current_line} {heading_stack:?}\",\n                                        )\n                                    }\n                                    test.precision_limits = code_block_info.precision_limits;\n                                    let _ = parser.next();\n                                    break;\n                                }\n                            }\n                        }\n\n                        // test must be over, continue at the top\n                        Some((Event::Start(CodeBlock(..)), _))\n                        | Some((Event::Start(Heading(..)), _)) => {\n                            tests.push(test);\n                            continue 'block_hunt;\n                        }\n\n                        // EOF, we're done\n                        None => {\n                            tests.push(test);\n                            break 'block_hunt;\n                        }\n\n                        // for now we allow text between the test and it's output\n                        // TODO should/can we forbid this?\n                        _ => {\n                            let _ = parser.next();\n                        }\n                    };\n                }\n\n                // consume the output\n                consume_text_until!(parser yields Event::End(CodeBlock(Fenced(..))) =>\n                    |text: CowStr| {\n                        let rows = text.split('\\n').skip(2).filter(|s| !s.is_empty()).map(|s|\n                            s.split('|').map(|s| s.trim().to_string()).collect::<Vec<_>>()\n                        );\n                        test.output.extend(rows);\n                    }\n                );\n\n                tests.push(test);\n            }\n\n            _ => (),\n        }\n    }\n    // create filename from full path\n    let file_name = Path::new(&file_stem).file_stem().unwrap().to_str().unwrap();\n    TestFile {\n        name: file_name.to_string(),\n        stateless,\n        tests,\n    }\n}\n\nstruct CodeBlockInfo {\n    kind: BlockKind,\n    transactional: bool,\n    ignore_output: bool,\n    precision_limits: HashMap<usize, usize>,\n    min_toolkit_version: Option<Version>,\n    creation: bool,\n    validation: bool,\n}\n\n#[derive(Clone, Copy)]\nenum BlockKind {\n    Sql,\n    Output,\n    Other,\n}\n\nfn parse_code_block_info(info: &str) -> CodeBlockInfo {\n    let tokens = info.split(',');\n\n    let mut info = CodeBlockInfo {\n        kind: BlockKind::Other,\n        transactional: true,\n        ignore_output: false,\n        precision_limits: HashMap::new(),\n        min_toolkit_version: None,\n        creation: false,\n        validation: false,\n    };\n\n    for token in tokens {\n        match token.trim() {\n            \"ignore\" => {\n                if let BlockKind::Sql = info.kind {\n                    info.kind = BlockKind::Other;\n                }\n            }\n            \"non-transactional\" => info.transactional = false,\n            \"ignore-output\" => info.ignore_output = true,\n            m if m.starts_with(\"min-toolkit-version\") => {\n                // TODO Can we assume that version is greater than 1.10.1 since current tests don't have a min version? This means we ccan skip edge cases of 1.4/1.10.0-dev/etc.\n                info.min_toolkit_version =\n                    Some(Version::parse(token.trim_start_matches(\"min-toolkit-version=\")).unwrap())\n            } // not great, shouldn't assume they typed in a valid version. fix later\n            \"creation\" => info.creation = true,\n            \"validation\" => info.validation = true,\n            \"output\" => info.kind = BlockKind::Output,\n            s if s.eq_ignore_ascii_case(\"sql\") => info.kind = BlockKind::Sql,\n            p if p.starts_with(\"precision\") => {\n                // syntax `precision(col: bytes)`\n                let precision_err =\n                    || -> ! { panic!(\"invalid syntax for `precision(col: bytes)` found `{p}`\") };\n                let arg = &p[\"precision\".len()..];\n                if arg.as_bytes().first() != Some(&b'(') || arg.as_bytes().last() != Some(&b')') {\n                    precision_err()\n                }\n                let arg = &arg[1..arg.len() - 1];\n                let args: Vec<_> = arg.split(':').collect();\n                if args.len() != 2 {\n                    precision_err()\n                }\n                let column = args[0].trim().parse().unwrap_or_else(|_| precision_err());\n                let length = args[1].trim().parse().unwrap_or_else(|_| precision_err());\n                let old = info.precision_limits.insert(column, length);\n                if old.is_some() {\n                    panic!(\"duplicate precision for column {column}\")\n                }\n            }\n            _ => {}\n        }\n    }\n\n    info\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashMap;\n\n    use semver::{BuildMetadata, Prerelease, Version};\n\n    #[test]\n    fn extract() {\n        use super::{Test, TestFile};\n\n        let file = r##\"\n# Test Parsing\n```SQL,creation\nselect * from foo\n```\n```output\n```\n\n```SQL,creation\nselect * from multiline\n```\n```output\n ?column?\n----------\n    value\n```\n\n## ignored\n```SQL,ignore,creation\nselect * from foo\n```\n\n## non-transactional,creation\n```SQL,non-transactional,creation\nselect * from bar\n```\n```output, precision(1: 3)\n a | b\n---+---\n 1 | 2\n```\n\n## no output\n```SQL,ignore-output,creation\nselect * from baz\n```\n\n## end by header\n```SQL,creation\nselect * from quz\n```\n\n## end by file\n```SQL,creation\nselect * from qat\n```\n\n## has a min-toolkit-version\n```SQL,creation,min-toolkit-version=1.10.1\nselect * from qat\n```\n\"##;\n\n        let tests = super::extract_tests_from_string(file, \"/test/file.md\");\n        let expected = TestFile {\n            name: \"file\".to_string(),\n            stateless: false,\n            tests: vec![\n                Test {\n                    location: \"/test/file.md:3\".to_string(),\n                    header: \"`Test Parsing`\".to_string(),\n                    text: \"select * from foo\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                    creation: true,\n                    min_toolkit_version: None,\n                    validation: false,\n                },\n                Test {\n                    location: \"/test/file.md:9\".to_string(),\n                    header: \"`Test Parsing`\".to_string(),\n                    text: \"select * from multiline\\n\".to_string(),\n                    output: vec![vec![\"value\".to_string()]],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                    creation: true,\n                    min_toolkit_version: None,\n                    validation: false,\n                },\n                Test {\n                    location: \"/test/file.md:24\".to_string(),\n                    header: \"`Test Parsing`::`non-transactional,creation`\".to_string(),\n                    text: \"select * from bar\\n\".to_string(),\n                    output: vec![vec![\"1\".to_string(), \"2\".to_string()]],\n                    transactional: false,\n                    ignore_output: false,\n                    precision_limits: [(1, 3)].iter().cloned().collect(),\n                    creation: true,\n                    min_toolkit_version: None,\n                    validation: false,\n                },\n                Test {\n                    location: \"/test/file.md:34\".to_string(),\n                    header: \"`Test Parsing`::`no output`\".to_string(),\n                    text: \"select * from baz\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: true,\n                    precision_limits: HashMap::new(),\n                    creation: true,\n                    min_toolkit_version: None,\n                    validation: false,\n                },\n                Test {\n                    location: \"/test/file.md:39\".to_string(),\n                    header: \"`Test Parsing`::`end by header`\".to_string(),\n                    text: \"select * from quz\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                    creation: true,\n                    min_toolkit_version: None,\n                    validation: false,\n                },\n                Test {\n                    location: \"/test/file.md:44\".to_string(),\n                    header: \"`Test Parsing`::`end by file`\".to_string(),\n                    text: \"select * from qat\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                    creation: true,\n                    min_toolkit_version: None,\n                    validation: false,\n                },\n                Test {\n                    location: \"/test/file.md:49\".to_string(),\n                    header: \"`Test Parsing`::`has a min-toolkit-version`\".to_string(),\n                    text: \"select * from qat\\n\".to_string(),\n                    output: vec![],\n                    transactional: true,\n                    ignore_output: false,\n                    precision_limits: HashMap::new(),\n                    creation: true,\n                    min_toolkit_version: Some(Version {\n                        major: 1,\n                        minor: 10,\n                        patch: 1,\n                        pre: Prerelease::EMPTY,\n                        build: BuildMetadata::EMPTY,\n                    }),\n                    validation: false,\n                },\n            ],\n        };\n        assert!(\n            tests == expected,\n            \"left: {:#?}\\n right: {:#?}\",\n            tests,\n            expected\n        );\n    }\n}\n"
  },
  {
    "path": "tools/update-tester/src/testrunner/stabilization.rs",
    "content": "pub use stabilization_info::*;\n\n#[path = \"../../../../extension/src/stabilization_info.rs\"]\nmod stabilization_info;\n\n#[macro_export]\nmacro_rules! functions_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($fn_name: ident ( $( $($fn_type: ident)+ ),* ) ),* $(,)?\n            }\n        )*\n    ) => {\n        pub static $export_symbol: &[&str] = &[\n            $(\n                $(stringify!( $fn_name( $( $($fn_type)+  ),* ) ),)*\n            )*\n        ];\n    };\n}\n\n#[macro_export]\nmacro_rules! types_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($type_name: ident),* $(,)?\n            }\n        )*\n    ) => {\n        pub static $export_symbol: &[&str] = &[\n            $(\n                $(stringify!($type_name),)*\n            )*\n        ];\n    };\n}\n\n#[macro_export]\nmacro_rules! operators_stabilized_at {\n    (\n        $export_symbol: ident\n        $(\n            $version: literal => {\n                $($operator_name: literal ( $( $($fn_type: ident)+ ),* ) ),* $(,)?\n            }\n        )*\n    ) => {\n        #[allow(non_snake_case)]\n        pub fn $export_symbol() -> std::collections::HashSet<String> {\n            static OPERATORS: &[(&str, &[&str])] = &[\n                $(\n                    $(\n                        (\n                            $operator_name,\n                            &[\n                                $( stringify!($($fn_type)+) ),*\n                            ]\n                        ),\n                    )*\n                )*\n            ];\n            OPERATORS.iter().map(|(name, types)| {\n                format!(\"{}({})\", name, types.join(\",\"))\n            }).collect()\n        }\n    };\n}\n"
  },
  {
    "path": "tools/update-tester/src/testrunner.rs",
    "content": "use colored::Colorize;\nuse semver::{BuildMetadata, Prerelease, Version};\n\nuse crate::{defer, parser, Deferred};\nuse postgres::{Client, NoTls, SimpleQueryMessage};\nuse postgres_connection_configuration::ConnectionConfig;\nmod stabilization;\n\nuse crate::parser::Test;\nuse postgres::error::DbError;\n\nuse std::{borrow::Cow, error::Error, fmt};\npub fn run_update_tests<OnErr: FnMut(Test, TestError)>(\n    root_config: &ConnectionConfig,\n    current_toolkit_version: String,\n    old_toolkit_versions: Vec<String>,\n    mut on_error: OnErr,\n) -> Result<(), xshell::Error> {\n    for old_toolkit_version in old_toolkit_versions {\n        eprintln!(\n            \" {} {old_toolkit_version} -> {current_toolkit_version}\",\n            \"Testing\".bold().cyan()\n        );\n\n        let test_db_name =\n            format!(\"tsdb_toolkit_test_{old_toolkit_version}--{current_toolkit_version}\");\n        let test_config = root_config.with_db(&test_db_name);\n        with_temporary_db(&test_db_name, root_config, || {\n            let mut test_client = connect_to(&test_config);\n\n            let errors = test_client\n                .create_test_objects_from_files(test_config, old_toolkit_version.clone());\n\n            for (test, error) in errors {\n                match error {\n                    Ok(..) => continue,\n                    Err(error) => on_error(test, error),\n                }\n            }\n\n            let errors = test_client\n                .validate_test_objects_from_files(test_config, old_toolkit_version.clone());\n\n            for (test, error) in errors {\n                match error {\n                    Ok(..) => continue,\n                    Err(error) => on_error(test, error),\n                }\n            }\n        });\n        eprintln!(\n            \"{} {old_toolkit_version} -> {current_toolkit_version}\",\n            \"Finished\".bold().green()\n        );\n    }\n    Ok(())\n}\n\npub fn create_test_objects_for_package_testing<OnErr: FnMut(Test, TestError)>(\n    root_config: &ConnectionConfig,\n    mut on_error: OnErr,\n) -> Result<(), xshell::Error> {\n    eprintln!(\" {}\", \"Creating test objects\".bold().cyan());\n\n    let test_db_name = \"tsdb_toolkit_test\";\n    let test_config = root_config.with_db(test_db_name);\n\n    let mut client = connect_to(root_config).0;\n\n    let drop = format!(r#\"DROP DATABASE IF EXISTS \"{test_db_name}\"\"#);\n    client\n        .simple_query(&drop)\n        .unwrap_or_else(|e| panic!(\"could not drop db {test_db_name} due to {e}\"));\n    let create = format!(r#\"create database \"{test_db_name}\"\"#);\n    client\n        .simple_query(&create)\n        .unwrap_or_else(|e| panic!(\"could not create db {test_db_name} due to {e}\"));\n\n    let mut test_client = connect_to(&test_config);\n\n    let create = \"CREATE EXTENSION timescaledb_toolkit\";\n    test_client\n        .simple_query(create)\n        .unwrap_or_else(|e| panic!(\"could not install extension due to {e}\",));\n\n    let current_toolkit_version = test_client.get_installed_extension_version();\n\n    // create test objects\n    let errors = test_client.create_test_objects_from_files(test_config, current_toolkit_version);\n\n    for (test, error) in errors {\n        match error {\n            Ok(..) => continue,\n            Err(error) => on_error(test, error),\n        }\n    }\n    eprintln!(\"{}\", \"Finished Object Creation\".bold().green());\n    Ok(())\n}\n\nfn connect_to(config: &ConnectionConfig<'_>) -> TestClient {\n    let client = Client::connect(&config.config_string(), NoTls).unwrap_or_else(|e| {\n        panic!(\n            \"could not connect to postgres DB {database} due to {e}\",\n            database = config.database.unwrap_or(\"\"),\n            e = e\n        )\n    });\n    TestClient(client)\n}\n\npub fn update_to_and_validate_new_toolkit_version<OnErr: FnMut(Test, TestError)>(\n    root_config: &ConnectionConfig,\n    mut on_error: OnErr,\n) -> Result<(), xshell::Error> {\n    // update extension to new version\n    let test_db_name = \"tsdb_toolkit_test\";\n    let test_config = root_config.with_db(test_db_name);\n\n    let mut test_client = connect_to(&test_config);\n\n    // get the currently installed version before updating\n    let old_toolkit_version = test_client.get_installed_extension_version();\n\n    test_client.update_to_current_toolkit_version();\n    // run validation tests\n    let errors = test_client.validate_test_objects_from_files(test_config, old_toolkit_version);\n\n    for (test, error) in errors {\n        match error {\n            Ok(..) => continue,\n            Err(error) => on_error(test, error),\n        }\n    }\n\n    // This close needs to happen before trying to drop the DB or else panics with `There is 1 other session using the database.`\n    test_client\n        .0\n        .close()\n        .unwrap_or_else(|e| panic!(\"Could not close connection to postgres DB due to {e}\"));\n    // if the validation passes, drop the db\n    let mut client = connect_to(root_config).0;\n    eprintln!(\"{}\", \"Dropping database.\".bold().green());\n\n    let drop = format!(r#\"DROP DATABASE IF EXISTS \"{test_db_name}\"\"#);\n    client\n        .simple_query(&drop)\n        .unwrap_or_else(|e| panic!(\"could not drop db {test_db_name} due to {e}\"));\n    Ok(())\n}\n\n//---------------//\n//- DB creation -//\n//---------------//\n\nfn with_temporary_db<T>(\n    db_name: impl AsRef<str>,\n    root_config: &ConnectionConfig<'_>,\n    f: impl FnOnce() -> T,\n) -> T {\n    let _db_dropper = create_db(root_config, db_name.as_ref());\n    let res = f();\n    drop(_db_dropper);\n    res\n}\n\n// create a database returning an guard that will DROP the db on `drop()`\n#[must_use]\nfn create_db<'a>(\n    root_config: &'a ConnectionConfig<'_>,\n    new_db_name: &'a str,\n) -> Deferred<(), impl FnMut() + 'a> {\n    let mut client = connect_to(root_config).0;\n    let create = format!(r#\"CREATE DATABASE \"{new_db_name}\"\"#);\n    client\n        .simple_query(&create)\n        .unwrap_or_else(|e| panic!(\"could not create db {new_db_name} due to {e}\"));\n\n    defer(move || {\n        let mut client = connect_to(root_config).0;\n        let drop = format!(r#\"DROP DATABASE \"{new_db_name}\"\"#);\n        client\n            .simple_query(&drop)\n            .unwrap_or_else(|e| panic!(\"could not drop db {new_db_name} due to {e}\"));\n    })\n}\n\n//-------------------//\n//- Test Components -//\n//-------------------//\n\nstruct TestClient(Client);\n\ntype QueryValues = Vec<Vec<Option<String>>>;\n\nimpl TestClient {\n    fn install_toolkit_at_version(&mut self, old_toolkit_version: &str) {\n        let create =\n            format!(r#\"CREATE EXTENSION timescaledb_toolkit VERSION \"{old_toolkit_version}\"\"#);\n        self.simple_query(&create).unwrap_or_else(|e| {\n            panic!(\"could not install extension at version {old_toolkit_version} due to {e}\",)\n        });\n    }\n\n    fn create_test_objects_from_files(\n        &mut self,\n        root_config: ConnectionConfig<'_>,\n        current_toolkit_version: String,\n    ) -> Vec<(Test, Result<(), TestError>)> {\n        let all_tests = parser::extract_tests(\"tests/update\");\n        // Hack to match previous versions of toolkit that don't conform to Semver.\n        let current_toolkit_semver = match current_toolkit_version.as_str() {\n            \"1.4\" => Version {\n                major: 1,\n                minor: 4,\n                patch: 0,\n                pre: Prerelease::EMPTY,\n                build: BuildMetadata::EMPTY,\n            },\n            \"1.5\" => Version {\n                major: 1,\n                minor: 5,\n                patch: 0,\n                pre: Prerelease::EMPTY,\n                build: BuildMetadata::EMPTY,\n            },\n            \"1.10.0-dev\" => Version {\n                major: 1,\n                minor: 10,\n                patch: 0,\n                pre: Prerelease::EMPTY,\n                build: BuildMetadata::EMPTY,\n            },\n            x => Version::parse(x).unwrap(),\n        };\n\n        let errors: Vec<_> = all_tests\n            .into_iter()\n            .flat_map(|tests| {\n                let mut db_creation_client = connect_to(&root_config);\n\n                let test_db_name = format!(\"{}_{}\", tests.name, current_toolkit_version);\n\n                let drop = format!(r#\"DROP DATABASE IF EXISTS \"{test_db_name}\"\"#);\n                db_creation_client\n                    .simple_query(&drop)\n                    .unwrap_or_else(|e| panic!(\"could not drop db {test_db_name} due to {e}\"));\n                let locale_flags = {\n                    match std::process::Command::new(\"locale\").arg(\"-a\").output() {\n                        Ok(cmd)\n                            if String::from_utf8_lossy(&cmd.stdout)\n                                .lines()\n                                .any(|l| l == \"C.UTF-8\") =>\n                        {\n                            \"LC_COLLATE 'C.UTF-8' LC_CTYPE 'C.UTF-8'\"\n                        }\n                        _ => \"LC_COLLATE 'C' LC_CTYPE 'C'\",\n                    }\n                };\n                let create = format!(r#\"CREATE DATABASE \"{test_db_name}\" {locale_flags}\"#,);\n                db_creation_client\n                    .simple_query(&create)\n                    .unwrap_or_else(|e| panic!(\"could not create db {test_db_name} due to {e}\"));\n\n                let test_config = root_config.with_db(&test_db_name);\n\n                let mut test_client = connect_to(&test_config);\n                test_client\n                    .simple_query(&format!(\n                        r#\"ALTER DATABASE \"{test_db_name}\" SET timezone TO 'UTC';\"#,\n                    ))\n                    .unwrap_or_else(|e| panic!(\"could not set time zone to UTC due to {e}\"));\n\n                // install new version and make sure it is correct\n                test_client.install_toolkit_at_version(&current_toolkit_version);\n                let installed_version = test_client.get_installed_extension_version();\n                assert_eq!(\n                    installed_version, current_toolkit_version,\n                    \"installed unexpected version\"\n                );\n\n                tests\n                    .tests\n                    .into_iter()\n                    .filter(|x| x.creation)\n                    .filter(|x| match &x.min_toolkit_version {\n                        Some(version) => version <= &current_toolkit_semver,\n                        None => true,\n                    })\n                    .map(move |test| {\n                        let output = run_test(&mut test_client, &test);\n                        (test, output)\n                    })\n            })\n            .collect();\n        errors\n    }\n\n    fn update_to_current_toolkit_version(&mut self) {\n        let update = \"ALTER EXTENSION timescaledb_toolkit UPDATE\";\n        self.simple_query(update)\n            .unwrap_or_else(|e| panic!(\"could not update extension due to {e}\"));\n    }\n\n    fn validate_test_objects_from_files(\n        &mut self,\n        root_config: ConnectionConfig<'_>,\n        old_toolkit_version: String,\n    ) -> Vec<(Test, Result<(), TestError>)> {\n        let all_tests = parser::extract_tests(\"tests/update\");\n\n        let old_toolkit_semver = match old_toolkit_version.as_str() {\n            \"1.4\" => Version {\n                major: 1,\n                minor: 4,\n                patch: 0,\n                pre: Prerelease::EMPTY,\n                build: BuildMetadata::EMPTY,\n            },\n            \"1.5\" => Version {\n                major: 1,\n                minor: 5,\n                patch: 0,\n                pre: Prerelease::EMPTY,\n                build: BuildMetadata::EMPTY,\n            },\n            \"1.10.0-dev\" => Version {\n                major: 1,\n                minor: 10,\n                patch: 0,\n                pre: Prerelease::EMPTY,\n                build: BuildMetadata::EMPTY,\n            },\n            x => Version::parse(x).unwrap(),\n        };\n        let errors: Vec<_> = all_tests\n            .into_iter()\n            .flat_map(|tests| {\n                let test_db_name = format!(\"{}_{}\", tests.name, old_toolkit_version);\n\n                let test_config = root_config.with_db(&test_db_name);\n\n                let mut test_client = connect_to(&test_config);\n\n                test_client.update_to_current_toolkit_version();\n                let new_toolkit_version = test_client.get_installed_extension_version();\n                test_client.check_no_references_to_the_old_binary_leaked(&new_toolkit_version);\n\n                test_client.validate_stable_objects_exist();\n\n                tests\n                    .tests\n                    .into_iter()\n                    .filter(|x| x.validation)\n                    .filter(|x| match &x.min_toolkit_version {\n                        Some(min_version) => min_version <= &old_toolkit_semver,\n                        None => true,\n                    })\n                    .map(move |test| {\n                        let output = run_test(&mut test_client, &test);\n                        // ensure that the DB is dropped after the client\n                        (test, output)\n                    })\n            })\n            .collect();\n        errors\n    }\n\n    fn check_no_references_to_the_old_binary_leaked(&mut self, current_toolkit_version: &str) {\n        let query_get_leaked_objects = format!(\n            \"SELECT pg_proc.proname \\\n            FROM pg_catalog.pg_proc \\\n            WHERE pg_proc.probin LIKE '$libdir/timescaledb_toolkit%' \\\n              AND pg_proc.probin <> '$libdir/timescaledb_toolkit-{current_toolkit_version}';\",\n        );\n        let leaks = self\n            .simple_query(&query_get_leaked_objects)\n            .unwrap_or_else(|e| panic!(\"could query the leaked objects due to {e}\"));\n        let leaks = get_values(leaks);\n        // flatten the list of returned objects for better output on errors\n        // it shouldn't change the result since each row only has a single\n        // non-null element anyway.\n        let leaks: Vec<String> = leaks\n            .into_iter()\n            .flat_map(Vec::into_iter)\n            .flatten()\n            .collect();\n        assert!(\n            leaks.is_empty(),\n            \"objects reference the old binary: {:#?}\",\n            &*leaks,\n        )\n    }\n\n    #[must_use]\n    fn get_installed_extension_version(&mut self) -> String {\n        let get_extension_version = \"\\\n            SELECT extversion \\\n            FROM pg_extension \\\n            WHERE extname = 'timescaledb_toolkit'\";\n        let updated_version = self\n            .simple_query(get_extension_version)\n            .unwrap_or_else(|e| panic!(\"could get updated extension version due to {e}\"));\n\n        get_values(updated_version)\n            .pop() // should have 1 row\n            .expect(\"no timescaledb_toolkit version\")\n            .pop() // row should have one value\n            .expect(\"no timescaledb_toolkit version\")\n            .expect(\"no timescaledb_toolkit version\")\n    }\n\n    pub(crate) fn validate_stable_objects_exist(&mut self) {\n        for function in stabilization::STABLE_FUNCTIONS {\n            let check_existence = format!(\"SELECT '{function}'::regprocedure;\");\n            self.simple_query(&check_existence)\n                .unwrap_or_else(|e| panic!(\"error checking function existence: {e}\"));\n        }\n\n        for ty in stabilization::STABLE_TYPES {\n            let check_existence = format!(\"SELECT '{ty}'::regtype;\");\n            self.simple_query(&check_existence)\n                .unwrap_or_else(|e| panic!(\"error checking type existence: {e}\"));\n        }\n\n        for operator in stabilization::STABLE_OPERATORS() {\n            let check_existence = format!(\"SELECT '{operator}'::regoperator;\");\n            self.simple_query(&check_existence)\n                .unwrap_or_else(|e| panic!(\"error checking operator existence: {e}\"));\n        }\n    }\n}\n\nimpl std::ops::Deref for TestClient {\n    type Target = Client;\n\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n\nimpl std::ops::DerefMut for TestClient {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut self.0\n    }\n}\n\n// translate a messages into their contained values\nfn get_values(query_results: Vec<SimpleQueryMessage>) -> QueryValues {\n    query_results\n        .into_iter()\n        .filter_map(|msg| match msg {\n            SimpleQueryMessage::CommandComplete(_) => None,\n            SimpleQueryMessage::Row(row) => {\n                let mut values = Vec::with_capacity(row.len());\n                for i in 0..row.len() {\n                    values.push(row.get(i).map(|s| s.to_string()))\n                }\n                Some(values)\n            }\n            _ => unreachable!(),\n        })\n        .collect()\n}\n\n// Functions below this line are originally from sql-doctester/src/runner.rs\n\npub fn validate_output(output: Vec<SimpleQueryMessage>, test: &Test) -> Result<(), TestError> {\n    use SimpleQueryMessage::*;\n\n    let mut rows = Vec::with_capacity(test.output.len());\n    for r in output {\n        match r {\n            Row(r) => {\n                let mut row: Vec<String> = Vec::with_capacity(r.len());\n                for i in 0..r.len() {\n                    row.push(r.get(i).unwrap_or(\"\").to_string())\n                }\n                rows.push(row);\n            }\n            CommandComplete(_) => {}\n            _ => unreachable!(),\n        }\n    }\n    let output_error = |header: &str| {\n        format!(\n            \"{}\\n{expected}\\n{}{}\\n\\n{received}\\n{}{}\\n\\n{delta}\\n{}\",\n            header,\n            stringify_table(&test.output),\n            format!(\"({} rows)\", test.output.len()).dimmed(),\n            stringify_table(&rows),\n            format!(\"({} rows)\", rows.len()).dimmed(),\n            stringify_delta(&test.output, &rows),\n            expected = \"Expected\".bold().blue(),\n            received = \"Received\".bold().blue(),\n            delta = \"Delta\".bold().blue(),\n        )\n    };\n\n    if test.output.len() != rows.len() {\n        return Err(TestError::OutputError(output_error(\n            \"output has a different number of rows than expected.\",\n        )));\n    }\n\n    fn clamp_len<'s>(mut col: &'s str, idx: usize, test: &Test) -> &'s str {\n        let max_len = test.precision_limits.get(&idx);\n        if let Some(&max_len) = max_len {\n            if col.len() > max_len {\n                col = &col[..max_len]\n            }\n        }\n        col\n    }\n\n    let all_eq = test.output.iter().zip(rows.iter()).all(|(out, row)| {\n        out.len() == row.len()\n            && out\n                .iter()\n                .zip(row.iter())\n                .enumerate()\n                .all(|(i, (o, r))| clamp_len(o, i, test) == clamp_len(r, i, test))\n    });\n    if !all_eq {\n        return Err(TestError::OutputError(output_error(\n            \"output has a different values than expected.\",\n        )));\n    }\n    Ok(())\n}\nfn stringify_table(table: &[Vec<String>]) -> String {\n    use std::{cmp::max, fmt::Write};\n    if table.is_empty() {\n        return \"---\".to_string();\n    }\n    let mut width = vec![0; table[0].len()];\n    for row in table {\n        // Ensure that we have width for every column\n        // TODO this shouldn't be needed, but sometimes is?\n        if width.len() < row.len() {\n            width.extend((0..row.len() - width.len()).map(|_| 0));\n        }\n        for (i, value) in row.iter().enumerate() {\n            width[i] = max(width[i], value.len())\n        }\n    }\n    let mut output = String::with_capacity(width.iter().sum::<usize>() + width.len() * 3);\n    for row in table {\n        for (i, value) in row.iter().enumerate() {\n            if i != 0 {\n                output.push_str(\" | \")\n            }\n            let _ = write!(&mut output, \"{:>width$}\", value, width = width[i]);\n        }\n        output.push('\\n')\n    }\n\n    output\n}\n\n#[allow(clippy::needless_range_loop)]\nfn stringify_delta(left: &[Vec<String>], right: &[Vec<String>]) -> String {\n    use std::{cmp::max, fmt::Write};\n\n    static EMPTY_ROW: Vec<String> = vec![];\n    static EMPTY_VAL: String = String::new();\n\n    let mut width = vec![\n        0;\n        max(\n            left.first().map(Vec::len).unwrap_or(0),\n            right.first().map(Vec::len).unwrap_or(0)\n        )\n    ];\n    let num_rows = max(left.len(), right.len());\n    for i in 0..num_rows {\n        let left = left.get(i).unwrap_or(&EMPTY_ROW);\n        let right = right.get(i).unwrap_or(&EMPTY_ROW);\n        let cols = max(left.len(), right.len());\n        for j in 0..cols {\n            let left = left.get(j).unwrap_or(&EMPTY_VAL);\n            let right = right.get(j).unwrap_or(&EMPTY_VAL);\n            if left == right {\n                width[j] = max(width[j], left.len())\n            } else {\n                width[j] = max(width[j], left.len() + right.len() + 2)\n            }\n        }\n    }\n    let mut output = String::with_capacity(width.iter().sum::<usize>() + width.len() * 3);\n    for i in 0..num_rows {\n        let left = left.get(i).unwrap_or(&EMPTY_ROW);\n        let right = right.get(i).unwrap_or(&EMPTY_ROW);\n        let cols = max(left.len(), right.len());\n        for j in 0..cols {\n            let left = left.get(j).unwrap_or(&EMPTY_VAL);\n            let right = right.get(j).unwrap_or(&EMPTY_VAL);\n            if j != 0 {\n                let _ = write!(&mut output, \" | \");\n            }\n            let (value, padding) = if left == right {\n                (left.to_string(), width[j] - left.len())\n            } else {\n                let padding = width[j] - (left.len() + right.len() + 2);\n                let value = format!(\n                    \"{}{}{}{}\",\n                    \"-\".magenta(),\n                    left.magenta(),\n                    \"+\".yellow(),\n                    right.yellow()\n                );\n                (value, padding)\n            };\n            // trick to ensure correct padding, the color characters are counted\n            // if done the normal way.\n            let _ = write!(&mut output, \"{:>padding$}{}\", \"\", value, padding = padding);\n        }\n        let _ = writeln!(&mut output);\n    }\n    output\n}\n\n#[derive(Debug)]\npub enum TestError {\n    PgError(postgres::Error),\n    OutputError(String),\n}\n\nimpl fmt::Display for TestError {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        match self {\n            TestError::PgError(error) => {\n                match error.source().and_then(|e| e.downcast_ref::<DbError>()) {\n                    Some(e) => {\n                        use postgres::error::ErrorPosition::*;\n                        let pos = match e.position() {\n                            Some(Original(pos)) => format!(\"At character {pos}\"),\n                            Some(Internal { position, query }) => {\n                                format!(\"In internal query `{query}` at {position}\")\n                            }\n                            None => String::new(),\n                        };\n                        write!(\n                            f,\n                            \"{}\\n{}\\n{}\\n{}\",\n                            \"Postgres Error:\".bold().red(),\n                            e,\n                            e.detail().unwrap_or(\"\"),\n                            pos,\n                        )\n                    }\n                    None => write!(f, \"{error}\"),\n                }\n            }\n            TestError::OutputError(err) => write!(f, \"{} {err}\", \"Error:\".bold().red()),\n        }\n    }\n}\n\nimpl From<postgres::Error> for TestError {\n    fn from(error: postgres::Error) -> Self {\n        TestError::PgError(error)\n    }\n}\n\nimpl TestError {\n    pub fn annotate_position<'s>(&self, sql: &'s str) -> Cow<'s, str> {\n        match self.location() {\n            None => sql.into(),\n            Some(pos) => format!(\n                \"{}{}{}\",\n                &sql[..pos as usize],\n                \"~>\".bright_red(),\n                &sql[pos as usize..],\n            )\n            .into(),\n        }\n    }\n\n    fn location(&self) -> Option<u32> {\n        use postgres::error::ErrorPosition::*;\n        match self {\n            TestError::OutputError(..) => None,\n            TestError::PgError(e) => match e\n                .source()\n                .and_then(|e| e.downcast_ref::<DbError>().and_then(DbError::position))\n            {\n                None => None,\n                Some(Internal { .. }) => None,\n                Some(Original(pos)) => Some(pos.saturating_sub(1)),\n            },\n        }\n    }\n}\n\nfn run_test(client: &mut Client, test: &Test) -> Result<(), TestError> {\n    let output = client.simple_query(&test.text)?;\n    validate_output(output, test)\n}\n"
  }
]